Spring Boot Open Telemetry Starter + Auto Exporter interaction

[pan-daniel]

Good morning!

I have been reading through the documentation trying to make our project work with Spring Boot Open Telemetry starter, but to no avail.

I have project that is configured as such (relevant parts from POM):

        <opentelemetry-instrumentation-bom.version>2.7.0</opentelemetry-instrumentation-bom.version>
        <exporter-auto.version>0.27.0-alpha</exporter-auto.version>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-aop</artifactId>
        </dependency>
        <dependency>
            <groupId>io.opentelemetry.instrumentation</groupId>
            <artifactId>opentelemetry-spring-boot-starter</artifactId>
        </dependency>
         <dependency>
            <groupId>com.google.cloud.opentelemetry</groupId>
            <artifactId>exporter-auto</artifactId>
            <version>${exporter-auto.version}</version>
        </dependency>
 
     <dependencyManagement>
        <dependencies>       
         <dependency>
                <groupId>io.opentelemetry.instrumentation</groupId>
                <artifactId>opentelemetry-instrumentation-bom</artifactId>
                <version>${opentelemetry-instrumentation-bom.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
         </dependencies>
    </dependencyManagement>
        

And here are our application.yml:

otel:
  traces:
    exporter: google_cloud_trace
  metrics:
    exporter: google_cloud_monitoring
  logs:
    exporter: none
  java:
    disabled:
      resource:
        providers: io.opentelemetry.instrumentation.resources.ProcessResourceProvider

Spring Boot version is 3.3.3. The application is deployed on the GKE.

The auto export of tracing is working flawlessly, so I cannot say anything about it.

However, the monitoring is constantly complaining in the container, with the following message:

com.google.api.gax.rpc.InvalidArgumentException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: One or more TimeSeries could not be written: The set of resource labels is incomplete. Missing labels: (container_name namespace_name pod_name).; The set of resource labels is incomplete. Missing labels: (container_name namespace_name pod_name).; The set of resource labels is incomplete. Missing labels: (container_name namespace_name pod_name).
	at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:92)
	at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:98)
	at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:66)
	at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97)
	at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:84)
	at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1130)
	at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1298)
	at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1059)
	at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:809)
	at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:568)
	at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:538)
	at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
	at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
	at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
	at com.google.api.gax.grpc.ChannelPool$ReleasingClientCall$1.onClose(ChannelPool.java:569)
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:564)
	at io.grpc.internal.ClientCallImpl.access$100(ClientCallImpl.java:72)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:729)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:710)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1570)
	Suppressed: com.google.api.gax.rpc.AsyncTaskException: Asynchronous task failed
		at com.google.api.gax.rpc.ApiExceptions.callAndTranslateApiException(ApiExceptions.java:57)
		at com.google.api.gax.rpc.UnaryCallable.call(UnaryCallable.java:112)
		at com.google.cloud.monitoring.v3.MetricServiceClient.createTimeSeries(MetricServiceClient.java:1905)
		at com.google.cloud.monitoring.v3.MetricServiceClient.createTimeSeries(MetricServiceClient.java:1833)
		at com.google.cloud.opentelemetry.metric.CloudMetricClientImpl.createTimeSeries(CloudMetricClientImpl.java:40)
		at com.google.cloud.opentelemetry.metric.InternalMetricExporter.createTimeSeriesBatch(InternalMetricExporter.java:217)
		at com.google.cloud.opentelemetry.metric.InternalMetricExporter.export(InternalMetricExporter.java:202)
		at com.google.cloud.opentelemetry.metric.GoogleCloudMetricExporter.export(GoogleCloudMetricExporter.java:90)
		at io.opentelemetry.sdk.metrics.export.PeriodicMetricReader$Scheduled.doRun(PeriodicMetricReader.java:167)
		at io.opentelemetry.sdk.metrics.export.PeriodicMetricReader$Scheduled.run(PeriodicMetricReader.java:153)
		at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
		at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358)
		at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
		... 3 common frames omitted
Caused by: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: One or more TimeSeries could not be written: The set of resource labels is incomplete. Missing labels: (container_name namespace_name pod_name).; The set of resource labels is incomplete. Missing labels: (container_name namespace_name pod_name).; The set of resource labels is incomplete. Missing labels: (container_name namespace_name pod_name).
	at io.grpc.Status.asRuntimeException(Status.java:533)
	... 14 common frames omitted

The obvious point in here is that the resource labels are incomplete, so I checked out containers and indeed we didn't have those, but using the downward API I filled these out. I can see that the resource has the correct labels now.

But still, no avail. Later on I have found this relevant piece of documentation here about this other library:
https://opentelemetry.io/docs/zero-code/java/agent/configuration/#enable-resource-providers-that-are-disabled-by-default

This documentation has shown me that my thinking was right and I needed the labels for the resource, which now I have. But even after activating the resource provider either with env variables or properties, this doesn't change the behaviour of the exporter.

In fact, I am not even sure if they play nicely together, or is the auto-exporter not taking the detected resources into consideration. My thinking right now is that yes, we have the labels in the resource of k8s_container, but I guess the resource can also be TimeSeries and they do not have those appended?

So! Could you please assist me or point me to the right direction? Basically I want to achieve with just the Spring Boot Open Telemetry starter and the auto exporters the metrics to work. With tracing it does.

Additionally, is there a way to limit what kind of spans does the exporter take into account? I followed the Spring Boot's documentation in disabling the actuator from being traced, but it seems also to have no effect on the exporter.

@Configuration
public class FilterPaths {

  @Bean
  public AutoConfigurationCustomizerProvider otelCustomizer() {
    return p ->
        p.addSamplerCustomizer(
            (fallback, config) ->
                RuleBasedRoutingSampler.builder(SpanKind.SERVER, fallback)
                    .drop(UrlAttributes.URL_PATH, "^/actuator")
                    .build());
  }
}

Thank you for your assistance!

[psx95]

Hi @pan-daniel,
Thank you for reporting this issue. I am looking at the error and it seems to me that the resource detection is not working.

The GCP Resource Detector is bundled with the OpenTelemetry Spring Boot Starter, but is disabled by default. You need to enable the GCP resource detector.

A few other things I noticed:

• The latest version for the exporter is v0.31.0-alpha.
• We recommend using the shaded version of the dependency when using the auto-exporter with the OpenTelemetry Agent (The advice is written for Java Agent, but I think it would apply to the Spring Boot Starter as well). The link showcases a maven code snippet of using shaded dependency.

Let me know if this helps.

[pan-daniel]

Hey!

Thanks for getting back to me.

After enabling the GCP resource detection again and upgrading the library to v0.31.0-alpha it works. I don't know how I missed that, I thought I was using the latest possible version of the exporter. Sorry for the overlook.

The recommendation of using the shaded version (latest) also did not work out, just fyi:

SLF4J: No SLF4J providers were found.
SLF4J: Defaulting to no-operation (NOP) logger implementation.
SLF4J: See https://www.slf4j.org/codes.html#noProviders for further details.

Would you however be able to advice about the path filtering as well from the original question?

Thank you a lot again

[psx95]

Hi @pan-daniel,

Regarding the No SLF4J providers were found. issue - this is not actually an error and it will go away if you add a dependency on a SLF4J implementation in your classpath. The reasoning and meaning behind this message is explained on the SLF4J manual.

Regarding your path filtering question - are you intentionally trying to drop some spans before export so that they don't show up?

Based on the config snippet you shared, it seems like you are trying to drop all traces that would be generated when visiting "/actuator". I have not used sampler customizer before, but I think the custom sampler you have used should work.

If it is not working, I suspect you are not supplying this customizer returned from your otelCustomizer() method to the AutoConfiguredOpenTelemetrySdk instance used by your application.

It is difficult to debug this further without looking at your code, but my suspicion is that because you want to customize the AutoConfiguredOpenTelemetrySdk instance used in your application that is instrumented using an agent (Spring Boot Starter Agent).
You need to create a custom agent extension to modify the AutoConfiguredOpenTelemetrySdk being used in your application. This is because when using an agent, you cannot modify the OpenTelemetry instance provided by the agent after your application starts.

You need to build an extension JAR in which you customize the sampling as you have shown above. This JAR would then be supplied through JVM args of you application: -Dotel.javaagent.extensions.

Note: The above information is based on my experience with Java AutoInstrumentation Agent, but I think Spring Boot Starter agent works in a similar fashion.
Alternatively, if you are using a OpenTelemetry Collector, you could filter spans in the collector using filterprocessor.

Let me know if this helps.

[psx95]

Closing this issue, please feel free to reopen in case there are any follow-up questions or concerns.

[psx95]

@pan-daniel, I looked into this a bit more and seems like Java Auto agent advice is not applicable to Spring boot starter.

As a quick follow-up: Are you using Spring Boot Native images ? If not, the default recommendation is to use OpenTelemetry Java Agent.

For Spring Boot Native images I have created a separate issue to document its support through an example in this repo: #376

Let me know if there are any additional questions.

[psx95]

Added sample Spring boot Application that uses OpenTelemetry Spring Starter Agent for auto-instrumentation. The sample can be located here.