OpenTelemetry JVM metrics are not properly mapped

[AlexanderWert]

The JVM metrics reported by OpenTelemetry Java agents are not properly mapped. In elastic/apm-server#8777 we changed the mapping to comply with the change in the metrics semantic convention, however, this mapping logic seems to be ignored.

The metric documents do appear under discover, however with wrong field names.

Example

This is how the metric document looks right now:

And this is how a valid JVM metrics document would look like:

So, seems that this mapping logic is not being applied.

Example data:

Here is some OTLP example data for the JVM metrics: https://gist.github.com/AlexanderWert/bf3b8a6cbbd02a345038bd8e8cac520f

Hypothesis:

Let's take a concrete metric: process.runtime.jvm.memory.usage

In this mapping logic it is assumed that this metric is reported as a Gauge metric type, however, in fact this metric (process.runtime.jvm.memory.usage) has the type sum (as we can see in the example data).
So very likely the root cause is the wrong metric type in the mapping logic.

Here is the OTel spec for the metrics: https://opentelemetry.io/docs/reference/specification/metrics/semantic_conventions/runtime-environment-metrics/#jvm-metrics

All types of counters (Counter, UpDownCounter) are mapped to the sum metric type in the OTLP protocol!
So we need to have the mapping in the MetricTypeSumif-branch.

[felixbarny]

As an alternative to mapping/duplicating the metrics, we could also have a dedicated visualization for OTel. Duplicating all metrics doesn't seem scalable in the long-term.
We could even think about aligning the metric names in our agents with the OTel metric names in the future.

[adamleantech]

I'm also hitting this issue which makes the OTel Java integration quite frustrating. I can potentially use the elastic agent instead in the short term but I'd like to have confidence that switching to OTel at a later date would be seamless. Please can this be considered when designing a solution?

[rcooper85]

Has this been released in version 8.8? As I can't see any mention of it in the release notes.

[SylvainJuge]

@rcooper85 this was solved in the UI by making the UI rely on the captured OTel metrics when they are available, thus with either OTel agents and Elastic agents you should get similar results in UI, however the data is expected to remain in the captured format.

[mielastic]

@rcooper85 this was solved in the UI by making the UI rely on the captured OTel metrics when they are available, thus with either OTel agents and Elastic agents you should get similar results in UI, however the data is expected to remain in the captured format.

This does not seem to be resolved. I'm running 8.11.1 and OTel Java Agent 1.32.0 and in APM app, "System memory usage", "Garbage collection per minute" and "Garbage collection time spent per minute" are not showing any data. Is this still not fixed?

[herrBez]

Hi,
During an analysis I could see that with the Exporter "logging" (i.e., stdout) the metric "process.runtime.jvm.gc.duration" is exported, while with the otlp exporter (i.e., by sending the data to APM-Server) it is not collected. Is it possible that APM-Server is doing some manipulation and/or discarding this metric for some reasons?

[SylvainJuge]

Hi @herrBez , the process.runtime.jvm.* prefix is the one we are using for testing thus it should be properly supported.

Can you provide which version of APM server and OpenTelemetry agent you are using ?

Please note that now the JVM metrics are now part of the semantic conventions and are stable with a different name, which also means:

• any existing mapping in apm-server (or apm-data component) will need to be updated to support this. In theory this should be transparent but needs to be checked.
• dashboards in kibana will need to be updated Update APM dashboards with stable JVM OTel metrics  kibana#174445
• agent needs to be run with an extra option to use the stable JVM metrics definition as explained in Mark JVM metrics stable open-telemetry/semantic-conventions#569

[herrBez]

Hi,
Thank you for the answer. I am using APM-Server on Elastic-Agent 8.11.3 and Opentelemetry javaagent 1.32.0 without the OTEL_SEMCONV_STABILITY_OPT_IN=jvm (TIL). With this configuration process.runtime.jvm.gc.duration (which should potentially work with my version of Kibana) is sent to APM-Server, but it is not indexed.

With the flag OTEL_SEMCONV_STABILITY_OPT_IN=jvm, I can see the GC Metric jvm.gc.duration without the process.runtime prefix which should work once elastic/kibana#174445 is fixed.