management.metrics.tags has been deprecated without a replacement working for all metrics

[mhalbritter]

In 3.2.0, we deprecated management.metrics.tags (MetricsProperties#getTags). The replacement for this is management.observations.key-values, but this property is only applied to metrics created through Observations. If creating metrics in a different way, management.observations.key-values is not applied to these metrics.

I currently see these ways forward:

• Un-deprecate management.metrics.tags - then we whould have two places to set common tags for metrics, depending how they get created, which is not great.
• Apply management.observations.key-values to metrics created without a Observation, too - then we would only have one way to set common tags for metrics, which is great. But the observations part implies that it should only apply to Observations. Or is a metric an Observation, too?
• Create a new property, e.g. management.observability.common-tags which applies to all observability things (Observation, metrics, traces). Then we have to deprecate the new management.observations.key-values and redirect users to management.observability.common-tags.

[wilkinsona]

We discussed this today and we're leaning towards the 3rd option although we have some reservations about the subtle difference between observation and observability. We'd like to discuss it some more before making a final decision.

[Tristan971]

Maybe worth a note in the release notes paragraph associated? (if editing those is possible)

Because in this sentence

this property is only applied to metrics created through Observations. If creating metrics in a different way [...]

"in a different way" includes all the "default" JVM metrics which we can probably expect that nearly-everyone cares about (ie it's not really a niche thing)

(also no opinion on what solution here is best, trusting you will do something nice 👍)

[mhalbritter]

I will fix that section of the release notes when we have made our decision.

[mhalbritter]

We decided to undeprecate management.metrics.tags for now, and we'll rethink about the whole tags (and key-values) situation in a next release.

[o-shevchenko]

Today to set a tag for metrics and traces, we should set two properties:

management:
  observations:
    key-values.custom_tag: tag # For all tags and some metrics
   metrics:
     tags.custom_tag: tag # For all metrics

Do we have a ticket we can track to simplify this configuration?

Does key-values.custom_tag work to set tags for logs? Does it work in the same way as OTEL_RESOURCE_ATTRIBUTES env var?

Thanks!

[bclozel]

@o-shevchenko I believe management.observations.key-values.spring=framework will add a spring=framework low cardinality KeyValue to all outgoing observations. This will be turned into tags for metrics and traces.

If you're seeing some metrics not getting this information, it probably means that the instrumentation is using the Metrics API. Spring Framework and many Spring projects are now only relying on the Observation API for instrumentation.

Maybe your application is instrumenting itself with the Metrics API? You can consider using Observation to get both metrics and traces with the same API. The only limitation is that the Observation API will only create timers. Other types (like gauges and counters) cannot be created with Observations.

Maybe your application is using a third-party instrumentation? In this case, you can check out the list of Observation-instrumented projects - maybe an Observation-variant already exists for your use case.