Feature: additional pod metadata

[prometherion]

Describe the feature

Let each Pod running in a Tenant Namespace to inherit a set of labels or annotations defined in the Tenant manifest.

What would the new user story look like?

As a Cluster Administrator, I want to perform traffic shaping in order to allow QoS from the Networking standpoint by labelling and annotating Tenant workloads (pods).

Expected behavior

By defining a set of additional metadata in the Tenant specificaiton each workload created in one of their Namespaces will be labelled or annotated with the required information.

The Tenant Owner doesn't need to know in advance the required labels, Capsule will inject them, and override any user-input to prevent any kind of escalation.

[prometherion]

We've been demanded to address traffic shaping (aka QoS, or bandwith management) from a Tenant level.

This could be done using a CNI network plugin, or the Cilium Bandwitdh Manager.

@MaxFedotov @oliverbaehler @bsctl although an overalap with other popular policy engines, I think this feature could be interesting to have it available in the Tenant definition and making it declarative.

Thoughts?

[bsctl]

it would be a very valuable feature, imho

[GiuseppeChiesa-TomTom]

I support this feature.

We also encounter the same use-case, since we use some other shared cluster components, such as logging agents that can take advantage of pod annotations to enrich the log informations.
For example, with Scalyr/Dataset, you can might set an annotation to define the "tenant" attribute on each pod logline (see: https://app.scalyr.com/help/scalyr-agent-k8s#attaching), and the information can be used to enrich the log entries in the logging backend.
This would be beneficial so that multiple tenants, can use the attributes to easy finding their logs.