feat: implement support for podOptions and additionalMetadata

[gchiesa]

This PR implements the support for additionalMetadata enforced on Pods, similarly to the additionalMetadata for services.

Rationale

In the context of soft-tenancy model, a practical use-case is represented by the need to enforce annotations / labels on Pods so that the information captured by shared cluster level services can be enriched with Tenant domain metadata (e.g. tenant-id, cost-center, etc.)

A practical use-case 1: Scalyr/Dataset Logs

Logging agents are deployed in a cluster governed by Capsule. Multiple tenants are present. Our requirement is to be able to query logs captured from pods part of a specific tenant.
Scalyr offers the possibility to enrich the logs captured from each Pod by leveraging annotations.

With this functionality Pod will be automatically annotated with the information decided by cluster administrator. This will offer the possibility to categorize/query logs by tenant attributes.

A practical use-case 2: Prometheus Metrics

Similarly to the use case above, also prometheus metrics can be enriched with tenant level attributes that will translate in meta labels.

With this functionality and using a proper Prometheus relabeling configuration the pod label/annotation can be translated in metric attributes attached to the specific pod.

How To

Cluster administrators can configure the annotation/labels for the Pods created under a specific tenancies by using the following:

apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
  name: oil
spec:
  owners:
    - name: alice
      kind: User
  podOptions:
    additionalMetadata:
      annotations:
        # example for scalyr logging
        log.config.scalyr.com/attributes.tenant-name: oil
      labels:
        tenant-name: oil
        tenant-cost-center: "12345"
        slack: #oil-notifications

References

closes #756

[netlify]

✅ Deploy Preview for capsule-documentation canceled.

Name	Link
🔨 Latest commit	c4fdcef
🔍 Latest deploy log	https://app.netlify.com/sites/capsule-documentation/deploys/655f5b39c0e18300089fd9ff

[oliverbaehler]

Thanks for your awesome pr, generally it looks good to me. We will have to discuss, if it would also make sense to implement #737 along-side with this feature. WDYT @prometherion

Please sign of all your commits.

[prometherion]

LGTM as a contribution, just let's try to address the comments! 🚀

[prometherion]

Thanks @gchiesa for nailing this, as we agreed I helped in some refactoring of your code, this is the stepping stone for the v0.5.0 release!