Tracing multiple applications in application servers

[fmhwong]

In an application server that supports multiple applications, we are using OpenTracing API in JAX-RS filters to automatic instrument applications. Each application has its own instance of Tracer. Implementations such as BraveTracer or JaegerTracer can be instantiated and configured with their own service names. Multiple tracer implementations can export spans to a single OpenTracing server.

In OpenTelemetry, created 2 instances of Tracer using

Tracer tracer1 = OpenTelemetry.getTracerProvider().get("app1");
Tracer tracer2 = OpenTelemetry.getTracerProvider().get("app2");

Created a SpanProcessor and SpanExporter using

ManagedChannel jaegerChannel = ManagedChannelBuilder.forAddress(ip, port).usePlaintext().build();
JaegerGrpcSpanExporter jaegerExporter = JaegerGrpcSpanExporter.newBuilder().setServiceName("app1")
                .setChannel(jaegerChannel).setDeadlineMs(30000).build();
OpenTelemetrySdk.getTracerProvider().addSpanProcessor(SimpleSpansProcessor.newBuilder(jaegerExporter).build());

In this case, spans from tracer1 and tracer2 got reported as "app1" in the backend because of the serviceName set in the SpanExporter. Creating another instance of SpanProcessor for "app2" to the same backend will result of duplication of span IDs in the backend.

[Oberon00]

This is an API bug IMHO. addSpanProcessor should be on the TracerProvider, not the tracer, as the list of span processors is shared between all tracers.

[Oberon00]

Related to the topic (but not that particular issue) is open-telemetry/opentelemetry-specification#335 "Consider adding a resource (semantic convention) to distinguish HTTP applications"

[fmhwong]

It seems JaegerGrpcSpanExporter is capable of exporting spans for a single service name only. If JaegerGrpcSpanExporter can get the name from the Tracer and use it as the service name, this will eliminate the need to have multiple Exporters.

[Oberon00]

The Tracer is not meant to represent a service. It just represents the reporting artifact name and version. I think there is currently no way to do this properly with a single class loader, but you can look at open-telemetry/oteps#78 and the aforementioned open-telemetry/opentelemetry-specification#335 to voice your opinion. IMHO the service should be taken from the resource by the JaegerGrpcSpanExporter (we already have service.name in https://github.com/open-telemetry/opentelemetry-specification/tree/master/specification/resource/semantic_conventions#service).

[bogdandrutu]

Related #350

[jkwatson]

This is an API bug IMHO. addSpanProcessor should be on the TracerProvider, not the tracer, as the list of span processors is shared between all tracers.

addSpanProcessor is already on the TracerSdkProvider. I'm not sure what you're referring to.

[Oberon00]

You are right, I misread the posted sample code. addSpanProcessor is where it should be 👍