Add missing processors to metrics pipeline

[tongpu]

We've seen that some labels that are present on logs are not present on metrics, add the k8sattributes and resource/add_environment processors to the metrics pipeline fixes this issue.

[dmitryax]

@tongpu which backend do send the metrics to?

The processors are omitted in the metrics pipeline on purpose.

• All the k8s related metrics coming kubeletstats, k8scluster and receiver_creator (with k8s observer) receivers already have the k8s attributes, so there is no need in additional processor. Metrics coming from hostmetrics receiver cannot be enriched with k8s attributes because they are out of k8s context.

• deployment.environment attribute as a concept in Observability Cloud is only applicable to traces and logs. Correlation with metrics is happening in signalfx exporter which is looking at traces coming through it. If you want this attribute on metrics in Splunk Platform backend, please add the condition so it's added only if splunk-otel-collector.platformMetricsEnabled.

[tongpu]

which backend do send the metrics to?

We're sending everything to Splunk.

The processors are omitted in the metrics pipeline on purpose.

I already thought so, mostly opened the PR to get your feedback on this.

• All the k8s related metrics coming kubeletstats, k8scluster and receiver_creator (with k8s observer) receivers already have the k8s attributes, so there is no need in additional processor. Metrics coming from hostmetrics receiver cannot be enriched with k8s attributes because they are out of k8s context.

I'm going to look further into that with our Splunk team.

• deployment.environment attribute as a concept in Observability Cloud is only applicable to traces and logs. Correlation with metrics is happening in signalfx exporter which is looking at traces coming through it. If you want this attribute on metrics in Splunk Platform backend, please add the condition so it's added only if splunk-otel-collector.platformMetricsEnabled.

So it would be okay if I limit this PR to add the environment processor when splunkPlatform.metricsEnabled is true?

[dmitryax]

We're sending everything to Splunk.

I was asking which Splunk destination, Splunk Observability or Splunk Platform? :)

So it would be okay if I limit this PR to add the environment processor when splunkPlatform.metricsEnabled is true?

Yes, it should be ok

[tongpu]

We're sending everything to Splunk.

I was asking which Splunk destination, Splunk Observability or Splunk Platform? :)

Currently Splunk Platform, but we're going to set up a PoC with Splunk Observability in the future.