Capture some pod labels by default

[mtwo]

A common use case that I've heard at customer meetings is applying a common set of pod labels as dimensions / annotations on metrics, traces, and logs. Users can custom-define these labels, and there are often too many to efficiently apply all of them, but it would be nice if the Collector captured some common / built-in ones. For example, Datadog's agent does a nice job of this: https://docs.datadoghq.com/agent/kubernetes/tag/?tab=containerizedagent

[mstumpfx]

Hi @mtwo

By default, at least for metrics, we do capture labels as signalfx properties that are synced onto resource UID dimensions.

For example, for pod labels we sync this onto the k8s.pod.uid dimension. In the signalfx product you should be able to navigate to the Menu --> Metrics --> Metadata Finder --> search for your k8s UID by typing k8s.pod.uid: $your_pod_uid and you should be able to find a list of properties synced onto this dimension.

The way this works in the code (since this is undocumented sort of I'll write some of this out) is that a receiver may implement a metadata exporter. The Signalfx exporter implements a consume metadata function which ends up exporting this metadata to our backend. This is enabled by default in our chart already.
In the k8s cluster receiver, each resource type is watched for updates and when updated specific sets of metadata is synced which you can view here. You can use that function as a jumping off point to see what properties we sync for various resources onto what dimensions.

Specifically for pod labels, you can see this is added here along with some other properties, such as the owners reference of the pod (e.g. Daemonset, StatefulSet ...)

We do not capture annotations at this time, that would be an enhancement to the k8sclusterreceiver in contrib.

[atoulme]

Closing this issue as inactive. Please reopen if this is still being worked on.