OpenTelemetryMeterRegistry is registered despite instrumentation being disabled

[gavlyukovskiy]

Describe the bug
I'm using opentelemetry-javaagent with my spring boot application and I started getting this error:

[otel.javaagent 2022-07-10 15:43:56:773 +0200] [restartedMain] WARN io.opentelemetry.ApiUsageLogging - Instrument name "xxxx.xxxxxxxxx.xxxxxxx.xx-xxxxxx-xxxxx.xxxxxxx.xxxxxxxx.xxxxxxxxxx" is invalid, returning noop instrument. Instrument names must consist of 63 or fewer characters including alphanumeric, _, ., -, and start with a letter.
java.lang.AssertionError
	at io.opentelemetry.javaagent.shaded.io.opentelemetry.api.internal.ValidationUtil.log(ValidationUtil.java:54)
	at io.opentelemetry.javaagent.shaded.io.opentelemetry.api.internal.ValidationUtil.checkValidInstrumentName(ValidationUtil.java:70)
	at io.opentelemetry.javaagent.shaded.io.opentelemetry.api.internal.ValidationUtil.checkValidInstrumentName(ValidationUtil.java:59)
	at io.opentelemetry.javaagent.shaded.io.opentelemetry.api.metrics.DefaultMeter.gaugeBuilder(DefaultMeter.java:60)
	at io.opentelemetry.micrometer1shim.OpenTelemetryGauge.<init>(OpenTelemetryGauge.java:40)
	at io.opentelemetry.micrometer1shim.OpenTelemetryMeterRegistry.newGauge(OpenTelemetryMeterRegistry.java:71)
	at io.micrometer.core.instrument.MeterRegistry.lambda$gauge$1(MeterRegistry.java:304)
	at io.micrometer.core.instrument.MeterRegistry.lambda$registerMeterIfNecessary$5(MeterRegistry.java:563)
	at io.micrometer.core.instrument.MeterRegistry.getOrCreateMeter(MeterRegistry.java:618)
	at io.micrometer.core.instrument.MeterRegistry.registerMeterIfNecessary(MeterRegistry.java:570)
	at io.micrometer.core.instrument.MeterRegistry.registerMeterIfNecessary(MeterRegistry.java:563)
	at io.micrometer.core.instrument.MeterRegistry.gauge(MeterRegistry.java:304)
	at io.micrometer.core.instrument.Gauge$Builder.register(Gauge.java:195)
	at io.micrometer.core.instrument.distribution.HistogramGauges.<init>(HistogramGauges.java:124)
	at io.micrometer.core.instrument.distribution.HistogramGauges.register(HistogramGauges.java:89)
	at io.micrometer.core.instrument.distribution.HistogramGauges.getHistogramGauges(HistogramGauges.java:53)
	at io.micrometer.core.instrument.distribution.HistogramGauges.registerWithCommonFormat(HistogramGauges.java:48)
	at io.opentelemetry.micrometer1shim.OpenTelemetryMeterRegistry.newTimer(OpenTelemetryMeterRegistry.java:111)
	at io.micrometer.core.instrument.MeterRegistry.lambda$timer$2(MeterRegistry.java:318)
	at io.micrometer.core.instrument.MeterRegistry.getOrCreateMeter(MeterRegistry.java:618)
	at io.micrometer.core.instrument.MeterRegistry.registerMeterIfNecessary(MeterRegistry.java:570)
	at io.micrometer.core.instrument.MeterRegistry.timer(MeterRegistry.java:316)
	at io.micrometer.core.instrument.Timer$Builder.register(Timer.java:399)
	at io.micrometer.core.instrument.composite.CompositeTimer.registerNewMeter(CompositeTimer.java:151)
	at io.micrometer.core.instrument.composite.CompositeTimer.registerNewMeter(CompositeTimer.java:31)
	at io.micrometer.core.instrument.composite.AbstractCompositeMeter.add(AbstractCompositeMeter.java:68)
	at java.base/java.lang.Iterable.forEach(Iterable.java:75)
	at java.base/java.util.Collections$SetFromMap.forEach(Collections.java:5700)
	at io.micrometer.core.instrument.composite.CompositeMeterRegistry.lambda$new$0(CompositeMeterRegistry.java:67)
	at io.micrometer.core.instrument.composite.CompositeMeterRegistry.lock(CompositeMeterRegistry.java:189)
	at io.micrometer.core.instrument.composite.CompositeMeterRegistry.lambda$new$1(CompositeMeterRegistry.java:67)
	at io.micrometer.core.instrument.MeterRegistry.getOrCreateMeter(MeterRegistry.java:627)
	at io.micrometer.core.instrument.MeterRegistry.registerMeterIfNecessary(MeterRegistry.java:570)
	at io.micrometer.core.instrument.MeterRegistry.timer(MeterRegistry.java:316)
	at io.micrometer.core.instrument.Timer$Builder.register(Timer.java:399)

From the error message I see that my metric name is too long for opentelemetry, however I'm using Micrometer with prometheus exporter and I'd like to opt-out from metrics part of it.

My configuration explicitly disables metrics exporter and I also tried to disable micrometer instrumentation:

otel.metrics.exporter=none
otel.instrumentation.micrometer.enabled=false

but I cannot get rid of the warning.

From debugging I see that OpenTelemetryMeterRegistry is registered in CompositeMeterRegistry, I assume it comes from
MetricsInstrumentation, but I might be wrong.

Steps to reproduce
Start spring-boot application with opentelemetry-javaagent that declares a metric with large name:

Timer.builder("xxxx.xxxxxxxxx.xxxxxxx.xx-xxxxxx-xxxxx.xxxxxxx.xxxxxxxx.xxxxxxxxxx").register(registry)

What did you expect to see?
OpenTelemetryMeterRegistry is not added to global registry when metrics export is disabled and/or micrometer instrumentation is disabled.

What did you see instead?
OpenTelemetryMeterRegistry is added to global registry and warns about invalid metrics.

What version are you using?
v1.15.0

Environment
Compiler: OpenJDK (Amazon Coretto) 17.0.1+12-LTS
OS: MacOS 12.3

Additional context
From debug logs I see that instrumentation was correctly disabled:

[otel.javaagent 2022-07-10 15:38:49:549 +0200] [main] DEBUG io.opentelemetry.javaagent.tooling.instrumentation.InstrumentationLoader - Loading instrumentation micrometer [class io.opentelemetry.javaagent.instrumentation.micrometer.v1_5.MicrometerInstrumentationModule]
[otel.javaagent 2022-07-10 15:38:49:549 +0200] [main] DEBUG io.opentelemetry.javaagent.extension.instrumentation.InstrumentationModule - Instrumentation micrometer is disabled

[gavlyukovskiy]

Dug more inside logs and turns out that OpenTelemetryMeterRegistry is actually registered from OpenTelemetryMeterRegistryAutoConfiguration, and that one can be disabled by adding

otel.instrumentation.spring-boot-actuator-autoconfigure.enabled=false

I can't say it's very obvious since this is not documented anywhere. Perhaps you can check if otel.metrics.exporter=none and not add this auto-configuration?

[hpuac]

Hey we stumbled across the same problem. The metrics exporter is set to "none" but we were also seeing the same warning about the instrument name length.
Adding the flag suggested by @gavlyukovskiy worked for us as well.
What is wondering me is that this flag is not printed when using the script that is supposed to find all config properties: https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/.github/scripts/find-all-config-properties.sh
And I also can't find it in code but I can confirm that it has an effect.
Note: We're using version 1.18.0.

[nrekretep]

Hi!
We are getting a similar warning from the otel java agent:

WARN io.opentelemetry.sdk.metrics.SdkMeter - Instrument name "spring.security.filterchains.WebAsyncManagerIntegrationFilter.before" is invalid, returning noop instrument. Instrument names must consist of 63 or fewer characters including alphanumeric, _, ., -, and start with a letter.

It seems that some metrics from autoinstrumented libraries cannot be transported via opentelemetry because the maximum length of the instrument's name is restricted to 63 characters for performance reasons.

Do I understand this correctly?

At least from our current experiences (and we are still at the starting point with otel) we can tell that the longest metric name in our prometheus tsdb which was ingested via otel java agent and otel collector has 63 characters (spring.security.filterchains.JwtTokenAuthenticationFilter.after).

Is there a way to get around this limitation from the user side or any plans in the future to shorten instrument names which are too long?

Or is this a limitation of which one has to be aware of before deciding to use otel?

Thx!

[trask]

Linking related spec issue: open-telemetry/opentelemetry-specification#3422

[jebeaudet]

FYI the otel.instrumentation.spring-boot-actuator-autoconfigure.enabled is matched here using the name from this class.

We ran into the issue described here #7360 and we needed to disable it to circumvent the issue.

[ViliusS]

@jebeaudet you can actually leave OTel Spring Boot instrumentation enabled but disable Micrometer Shim with otel.springboot.micrometer.enabled=false

This should fix the metrics issue without disabling Spring Boot Auto-Instrumentation all together.

[jebeaudet]

@ViliusS Thanks for the suggestion. However, in my case, this shim config class didn't match in the first place because @ConditionalOnBean(Clock.class) didn't match. I had to go with otel.instrumentation.spring-boot-actuator-autoconfigure.enabled.