SDK AutoConfigure not working when switching from logging to otlp exporter

[ptabasso2]

Describe the bug
Hitting an issue when using the SDK AutoConfigure with the otlp exporter.
This has been observed when following the getting started guide instructions
(Getting started - Manual instrumentation)

Steps to reproduce
What works:

Using the logging exporter works well, the application starts and prints the expected output.

env OTEL_SERVICE_NAME=dice-server OTEL_TRACES_EXPORTER=logging OTEL_METRICS_EXPORTER=logging OTEL_LOGS_EXPORTER=logging java -jar ./build/libs/springdice-0.0.1-SNAPSHOT.jar

What does not work:

Using the otlp exporter makes the application failing during bootstrap with the exception below:

env OTEL_SERVICE_NAME=dice-server OTEL_TRACES_EXPORTER=otlp OTEL_METRICS_EXPORTER=otlp OTEL_LOGS_EXPORTER=otlp java -jar ./build/libs/springdice-0.0.1-SNAPSHOT.jar

What did you expect to see?
I would have expected to see the spring boot app starting and sending traces to the otel collector when issuing a curl command against the controller endpoint

What did you see instead?
The app crashes during startup with the following exception:

2023-11-27T20:46:18.727Z  WARN 340140 --- [           main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'rollController' defined in URL [jar:nested:/root/springdice/build/libs/springdice-0.0.1-SNAPSHOT.jar/!BOOT-INF/classes/!/com/example/springdice/RollController.class]: Unsatisfied dependency expressed through constructor parameter 0: Error creating bean with name 'openTelemetry' defined in com.example.springdice.DiceApplication: Failed to instantiate [io.opentelemetry.api.OpenTelemetry]: Factory method 'openTelemetry' threw exception with message: 'io.opentelemetry.exporter.internal.grpc.GrpcExporterBuilder io.opentelemetry.exporter.internal.grpc.GrpcExporterBuilder.setMeterProvider(java.util.function.Supplier)'
2023-11-27T20:46:18.732Z  INFO 340140 --- [           main] o.apache.catalina.core.StandardService   : Stopping service [Tomcat]
2023-11-27T20:46:18.756Z  INFO 340140 --- [           main] .s.b.a.l.ConditionEvaluationReportLogger : 

Error starting ApplicationContext. To display the condition evaluation report re-run your application with 'debug' enabled.
2023-11-27T20:46:18.792Z ERROR 340140 --- [           main] o.s.b.d.LoggingFailureAnalysisReporter   : 

***************************
APPLICATION FAILED TO START
***************************

Description:

An attempt was made to call a method that does not exist. The attempt was made from the following location:

    io.opentelemetry.exporter.otlp.trace.OtlpGrpcSpanExporterBuilder.setMeterProvider(OtlpGrpcSpanExporterBuilder.java:193)

The following method did not exist:

    'io.opentelemetry.exporter.internal.grpc.GrpcExporterBuilder io.opentelemetry.exporter.internal.grpc.GrpcExporterBuilder.setMeterProvider(java.util.function.Supplier)'

The calling method's class, io.opentelemetry.exporter.otlp.trace.OtlpGrpcSpanExporterBuilder, was loaded from the following location:

    jar:nested:/root/springdice/build/libs/springdice-0.0.1-SNAPSHOT.jar/!BOOT-INF/lib/opentelemetry-exporter-otlp-1.32.0.jar!/io/opentelemetry/exporter/otlp/trace/OtlpGrpcSpanExporterBuilder.class

The called method's class, io.opentelemetry.exporter.internal.grpc.GrpcExporterBuilder, is available from the following locations:

    jar:nested:/root/springdice/build/libs/springdice-0.0.1-SNAPSHOT.jar/!BOOT-INF/lib/opentelemetry-exporter-common-1.31.0.jar!/io/opentelemetry/exporter/internal/grpc/GrpcExporterBuilder.class

The called method's class hierarchy was loaded from the following locations:

    io.opentelemetry.exporter.internal.grpc.GrpcExporterBuilder: jar:nested:/root/springdice/build/libs/springdice-0.0.1-SNAPSHOT.jar/!BOOT-INF/lib/opentelemetry-exporter-common-1.31.0.jar!/


Action:

Correct the classpath of your application so that it contains compatible versions of the classes io.opentelemetry.exporter.otlp.trace.OtlpGrpcSpanExporterBuilder and io.opentelemetry.exporter.internal.grpc.GrpcExporterBuilder

What version and what artifacts are you using?

dependencies {
	implementation("org.springframework.boot:spring-boot-starter-web")
	implementation("io.opentelemetry:opentelemetry-api:1.32.0")
	implementation("io.opentelemetry:opentelemetry-sdk:1.32.0")
	implementation("io.opentelemetry:opentelemetry-sdk-metrics:1.32.0")
	implementation("io.opentelemetry:opentelemetry-exporter-logging:1.32.0")
	implementation("io.opentelemetry:opentelemetry-semconv:1.23.0-alpha")
	implementation("io.opentelemetry:opentelemetry-sdk-extension-autoconfigure:1.32.0")
	implementation("io.opentelemetry:opentelemetry-sdk-extension-autoconfigure-spi:1.32.0")
	implementation("io.opentelemetry:opentelemetry-exporter-otlp:1.32.0")

}

Environment

Compiler + runtime

openjdk version "17.0.9" 2023-10-17 LTS
OpenJDK Runtime Environment Zulu17.46+19-CA (build 17.0.9+8-LTS)
OpenJDK 64-Bit Server VM Zulu17.46+19-CA (build 17.0.9+8-LTS, mixed mode, sharing)

Build tool:

------------------------------------------------------------
Gradle 8.4
------------------------------------------------------------

Build time:   2023-10-04 20:52:13 UTC
Revision:     e9251e572c9bd1d01e503a0dfdf43aedaeecdc3f

Kotlin:       1.9.10
Groovy:       3.0.17
Ant:          Apache Ant(TM) version 1.10.13 compiled on January 4 2023
JVM:          17.0.9 (Azul Systems, Inc. 17.0.9+8-LTS)
OS:           Linux 6.5.0-1008-gcp amd64 / Ubuntu 23.10

Additional context

It seems that the issue is tied to the opentelemetry-exporter-otlp-1.32.0.jar artifact that adds opentelemetry-exporter-common-1.31.0.jar (transitive dependency) instead of version 1.32.0.

The OtlpGrpcSpanExporterBuilder class which is provided by the opentelemetry-exporter-otlp-1.32.0.jar jar invokes the setMeterProvider() method and expects a supplier as an argument.

The 1.32.0 GrpcExporterBuilder class implementation exposes the setMeterProvider with the right signature (Supplier provided)

...
public GrpcExporterBuilder<T> setMeterProvider(Supplier<MeterProvider> meterProviderSupplier) {
        this.meterProviderSupplier = meterProviderSupplier;
        return this;
}
...

But the 1.31.0 GrpcExporterBuilder class implementation provides a simple MeterProvider type instead of a supplier which causes the issue

    public GrpcExporterBuilder<T> setMeterProvider(MeterProvider meterProvider) {
        this.meterProviderSupplier = () -> {
            return meterProvider;
        };
        return this;
    }

[jack-berg]

It seems that the issue is tied to the opentelemetry-exporter-otlp-1.32.0.jar artifact that adds opentelemetry-exporter-common-1.31.0.jar (transitive dependency) instead of version 1.32.0.

That should not be true. See the POM from opentelemetry-exporter-otlp:1.32.0 here: https://repo1.maven.org/maven2/io/opentelemetry/opentelemetry-exporter-otlp/1.32.0/opentelemetry-exporter-otlp-1.32.0.pom

Maybe something else is causing the dependency conflict? I know I've seen issues in the past with the gradle spring dependency management plugin overriding the version of OpenTelemetry specified by the user. See here that the 3.2.0 version of spring boot depends on version 1.31.0 of OpenTelemetry java, which is suspiciously the same version you're seeing resolved.

[ptabasso2]

Thanks for the prompt reply @jack-berg
It looks indeed that when using spring 2.x.x the versions are all aligned as expected for the artifacts.

I've just tested various versions of spring 3 and it seems that in this case for a given artifact you never get 1.32.0 for the dependent ones. They even vary as you change from 3.0.0, 3.0.6, 3.0.13 etc...Not sure exactly why but I'll dig deeper to figure out.

Thank you

[jack-berg]

I've actually come across this before. I couldn't find a way to get gradle dependencies to respect the versions from the opentelemetry-bom when using the spring boot gradle dependency management plugin. My solution was to bail on the spring boot dependency management plugin and add a dependency directly on the spring boot bom instead: newrelic/newrelic-opentelemetry-examples@4657142#diff-21bc7265f74c0805ef365c54fd9e2696b565e660a8f7237ecd7eb2c226b4b6b5

[ptabasso2]

Interesting. I'm just wondering if this couldn't be related to the gradle plugin itself for that particular version of spring. I'll try using maven to verify and also give it a shot by downgrading the gradle version. I'll keep you posted. Thanks for all your help so far!

[ptabasso2]

Hey @jack-berg

Just following up on this.
Not a solution but rather a workaround I'm using in my gradle script when using version 3.x
I would add the following block to make sure I'm getting 1.32.0 for any artifacts coming through transitive dependency

Worth noting that 3 of them are still in alpha (opentelemetry-semconv, opentelemetry-api-events, opentelemetry-extension-incubator ) hence the exception below

configurations.all {
	resolutionStrategy.eachDependency {
		if (requested.group == "io.opentelemetry" && requested.name !in listOf("opentelemetry-semconv", "opentelemetry-api-events", "opentelemetry-extension-incubator")) {
			useVersion("1.32.0")

		}
	}
} 

[PerfectSlayer]

Hey @jack-berg 👋
Bruce from Datadog, you might have noticed me during the Thursday Java SIG meetings.

@ptabasso2 reach me out about the issue and I start investigating it.

Issue Investigation

Spring Boot uses a Gradle plugin to manage dependencies: the Dependency Management Plugin.
So when you follow the OTel manual instrumentation documentation, you end up in the situation described above, not picking the expected version of the OTel dependencies.

I found two options to fix it:

1. Either removing the Spring dependency management plugin and everything will get back as expected. Spring might not be happy about the versions it needs though,
2. Or (better) Importing the OTel Maven BOM and declare dependencies without version number. It is the Spring way of handling dependencies:

dependencyManagement {
     imports {
          mavenBom 'io.opentelemetry:opentelemetry-bom:1.32.0'
     }
}

dependencies {
    implementation("org.springframework.boot:spring-boot-starter-web");
    implementation("io.opentelemetry:opentelemetry-api");
    implementation("io.opentelemetry:opentelemetry-sdk");
    implementation("io.opentelemetry:opentelemetry-sdk-metrics");
    implementation("io.opentelemetry:opentelemetry-exporter-logging");
    implementation("io.opentelemetry.semconv:opentelemetry-semconv:1.23.1-alpha")
    implementation("io.opentelemetry:opentelemetry-sdk-extension-autoconfigure");
    implementation("io.opentelemetry:opentelemetry-sdk-extension-autoconfigure-spi");
}

Using this configuration will properly include the expected OTel dependencies at compile and runtime.

Note

I think we can rename the issue to point that it is a Spring+Gradle specific issue. It will help other developers to find the workaround.

About the documentation page

As Spring is a popular framework, I think this detail should be mentioned in the documentation page to avoid future (hard-to-diagnosticate) issues.
I understand it is a Spring specific step that might be confusing for the non-Spring users but the sample is based on Spring.

And by the way, there is an error in the manual instrumentation page where is shows to import semconv 1.32.0-alpha which is absent from Maven Central (1.30.1-alpha at best).

    implementation("io.opentelemetry:opentelemetry-semconv:1.32.0-alpha");

Would you like me to have a PR about those two points?

[jkwatson]

the semconv doc needs to be updated to pointed to the new semconv artifact from the new semantic conventions repo, FYI:

implementation("io.opentelemetry.semconv:opentelemetry-semconv:1.23.1-alpha");

[trask]

Spring Boot uses a Gradle plugin to manage dependencies: the Dependency Management Plugin.
So when you follow the OTel manual instrumentation documentation, you end up in the situation described above, not picking the expected version of the OTel dependencies.

linking to related open-telemetry/opentelemetry.io#3613

and yes, doc help is always appreciated!

[taisph]

I may be facing a similar issue with a maven build. Since 1.22 I can only have one exporter dependency added in a build. Ie. if my pom.xml has

  <dependencyManagement>
    <dependencies>
      <dependency>
        <groupId>io.opentelemetry</groupId>
        <artifactId>opentelemetry-bom</artifactId>
        <version>1.34.1</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>
    </dependencies>
  </dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>io.opentelemetry</groupId>
      <artifactId>opentelemetry-exporter-otlp</artifactId>
    </dependency>
    <dependency>
      <groupId>io.opentelemetry</groupId>
      <artifactId>opentelemetry-exporter-logging</artifactId>
    </dependency>
  <!-- ... -->
  </dependencies>

and I use OTEL_METRICS_EXPORTER=otlp and OTEL_TRACES_EXPORTER=logging then I get the error:

Exception in thread "main" io.opentelemetry.sdk.autoconfigure.spi.ConfigurationException: otel.traces.exporter set to "logging" but opentelemetry-exporter-logging not found on classpath. Make sure to add it as a dependency.

If I switch the dependencies around, adding logging before otlp I get:

Exception in thread "main" io.opentelemetry.sdk.autoconfigure.spi.ConfigurationException: otel.metrics.exporter set to "otlp" but opentelemetry-exporter-otlp not found on classpath. Make sure to add it as a dependency.

[jack-berg]

Can you check that the versions of all your dependencies are aligned and if so, post a small reproduction app? The behavior you're reporting is not expected, and I believe there are unit tests which confirm that.

[taisph]

Can you check that the versions of all your dependencies are aligned and if so, post a small reproduction app? The behavior you're reporting is not expected, and I believe there are unit tests which confirm that.

I believe they were but now I've yanked out the manual instrumentation code and switched to using the agent instead. Less detailed than what I had before but seems to behave with both exporter types.