Spring Boot: Error when configuring OTLP exporter in application.properties

[joaopgrassi]

Describe the bug

I generated a Spring Boot app from start.spring.io, and followed the steps on the website to add the OTel dependencies (gradle) https://opentelemetry.io/docs/languages/java/automatic/spring-boot/#opentelemetry-spring-boot-starter

I then saw we can use the same SDK configuration inside application.properties. So I added:

otel.service.name=java.springboot.api
otel.exporter.otlp.endpoint=http://my-collector:4317

And then when I start the application, I get a lot of errors such as:

java.io.IOException: unexpected end of stream on http://my-collector:4317/...
	at okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.kt:210) ~[okhttp-4.12.0.jar:na]
	at okhttp3.internal.connection.Exchange.readResponseHeaders(Exchange.kt:110) ~[okhttp-4.12.0.jar:na]
	at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.kt:93) ~[okhttp-4.12.0.jar:na]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.12.0.jar:na]
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:34) ~[okhttp-4.12.0.jar:na]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.12.0.jar:na]
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95) ~[okhttp-4.12.0.jar:na]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.12.0.jar:na]
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83) ~[okhttp-4.12.0.jar:na]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.12.0.jar:na]
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76) ~[okhttp-4.12.0.jar:na]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.12.0.jar:na]
	at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201) ~[okhttp-4.12.0.jar:na]
	at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:517) ~[okhttp-4.12.0.jar:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[na:na]
	at java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]
	Suppressed: java.io.IOException: unexpected end of stream on http://localhost:4317/...
		... 17 common frames omitted
	Caused by: java.io.EOFException: \n not found: limit=15 content=000006040000000000000500004000…

But when I configure the OTLP exporter endpoint like this:

spring.otel.exporter.otlp.endpoint=http://my-collector:4317

It works, and I get the spans in my collector.

Funny thing, is that otel.service.name works. If I change that to spring.otel.service.name then I don't have the service name on the resource.

Steps to reproduce

• Generate a Spring Boot web app like: https://start.spring.io/#!type=gradle-project&language=java&platformVersion=3.2.4&packaging=jar&jvmVersion=17&groupId=com.example&artifactId=demo&name=demo&description=Demo%20project%20for%20Spring%20Boot&packageName=com.example.demo&dependencies=web

• Add the OTel dependencies like in the website https://opentelemetry.io/docs/languages/java/automatic/spring-boot/#dependency-management

• Add a application.properties containing:

otel.service.name=java.springboot.api
otel.exporter.otlp.endpoint=http://localhost:4317

• Run the app and see the errors (and no spans in the collector)

Expected behavior

The configuration works with the snippets from the OTel website

Actual behavior

No spans arriving when I use the configuration from the website instructions

Javaagent or library instrumentation version

opentelemetry-bom:1.36.0
opentelemetry-instrumentation-bom-alpha:2.2.0-alpha

Environment

JDK: eclipse-temurin:21-jdk
OS: Linux

Additional context

No response

[laurit]

Probably fails because the default otlp protocol is http/protobuf but you are sending to grpc endpoint (port 4317 is the default port for grpc, 4318 for http/protobuf). spring.otel.exporter.otlp.endpoint works because it is an unrecognized property and the default http/protobuf endpoint is used.

[joaopgrassi]

Are you sure? Because here: https://github.com/open-telemetry/opentelemetry-java/blob/main/sdk-extensions/autoconfigure/README.md#otlp-exporter-span-metric-and-log-exporters otel.exporter.otlp.protocol the default is grpc. 🤔

Edit: That was it, you are right @laurit. I guess the the docs here need to be updated then :

The transport protocol to use on OTLP trace, metric, and log requests. Options include grpc and http/protobuf. Default is grpc.

[laurit]

I guess the the docs here need to be updated

The referenced documentation is correct, for opentelemetry-java the default is grpc while opentelemetry-java-instrumentation defaults to http/protobuf. In opentelemetry-java-instrumentation the default was changed from grpc to http/protobuf with the 2.0 release. opentelemetry-java is still 1.x and the default there can't be changed without bumping the major version because it is a breaking change.

[joaopgrassi]

Hum, I understand what you mean, but I don't think the documentation is right. My flow was:

1. Go to OTel website to learn about OTel and Spring boot
2. Then I went to Configuration to learn how to configure the OTLP Exporter URI
3. The website then says:

The OpenTelemetry Starter supports all the SDK Autoconfiguration (since 2.2.0). You can set properties in the application.properties or the application.yaml file, or use environment variables.

4. I navigate to SDK Autoconfiguration
5. There I click on OTLP exporter (span, metric, and log exporters)
6. I arrive at https://github.com/open-telemetry/opentelemetry-java/blob/main/sdk-extensions/autoconfigure/README.md#otlp-exporter-span-metric-and-log-exporters, which tells me for otel.exporter.otlp.protocol grpc is the default.

I think the instructions then need to be modified to alert about the changes in instrumentation 2.0. I couldn't find this anywhere :(.

[zeitlinger]

That's right - I think this note got lost.

[laurit]

@zeitlinger can this issue be closed?

[zeitlinger]

Yes, the docs have been updated here: https://opentelemetry.io/docs/languages/java/automatic/configuration/

[perfumescent]

thanks guys, the doc is a little bit confusing. I've checked for multiple times and still considered java agent jar as using default grpc.

[zeitlinger]

thanks guys, the doc is a little bit confusing. I've checked for multiple times and still considered java agent jar as using default grpc.

can you point out what caused (which page) caused the confusion?

[perfumescent]

thanks guys, the doc is a little bit confusing. I've checked for multiple times and still considered java agent jar as using default grpc.

can you point out what caused (which page) caused the confusion?

OK, it's a kind of long road.

I started from multiple APM platforms and thus found opentelemetry. All of their docs mentioned configs like
-Dotel.exporter.otlp.endpoint=http://localhost:4317 -Dotel.exporter.otlp.headers. But I failed to run it with java program.

So I directly went for this page https://opentelemetry.io/docs/zero-code/java/agent/configuration/ as it is labeled as Java Agent.
However I couldnt find anything like otel.exporter.otlp config specification in this zero-code/java/agent/configuration unit. This is confusing.

Then I found all things with otel.exporter.otlp at https://opentelemetry.io/docs/languages/java/configuration/. I felt finally released and successfully confirmed that i was surely using the default protocol grpc.

This is confusing because java agent does not share the same config with java sdk while there is no noticeable specification on java agent page.

Is it clearly described?

[zeitlinger]

Yes, this is great feedback for our project to improve the docs: open-telemetry/opentelemetry.io#4881