Baggage propogation not working in Java application with auto instrumentation enabled

[vishalgoel1988]

Hi,

We have enabled auto instrumentation in one of our Java application. We are using opentelemetry-javaagent-all-1.6.2.jar for this purpose. This Java app exposes REST endpoints (Jersey over Jetty web server).
We tried calling REST endpoint from Postman and traces are collected fine. It is working and so far so good.

Our problem is we are unable to propogate baggage information. We want to have some key-value pairs that should get propogated from client to server (Java app) and further also in every span. For this, we tried below but no success in any of them:

• Tried to send "baggage" header from postman but it did not come as part of trace log and not even in Span object (while debugging)
  https://www.w3.org/TR/baggage/
• We set env vars while starting app as per below and passed those headers in the postman request but this also did not work.
  https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/docs/config/common.md#capturing-http-request-and-response-headers

Are we doing anything wrong here? What is the correct way to propogate baggage or some common information across spans?
Please help.

[mateuszrzeszutek]

Hey @vishalgoel1988 ,

• We set env vars while starting app as per below and passed those headers in the postman request but this also did not work.
  https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/docs/config/common.md#capturing-http-request-and-response-headers

This feature is not present in the 1.6.x versions - it will become a part of the 1.7.0 release. The docs you can see on the main branch describe the current development version; you can find the 1.6.2 documentation here.

Regarding baggage: baggage is not added automatically to spans. It definitely should be propagated and accessible by e.g. calling Baggage.current(). Can you try doing that in your code?

We want to have some key-value pairs that should get propogated from client to server (Java app) and further also in every span.

If you want to add baggage entries as span attributes to every span, you can implement a custom SpanProcessor that does that. You'll need to build a custom extension jar here to register that with the javaagent though.

[mateuszrzeszutek]

Do we have the timeline for 1.7.0 release. It will greatly help us in planning.

It should happen either this or in the next week.

Also, we tried Baggage.current() but it did not work. From postman, we are passing "baggage" header and in our code at the entry point, we are debugging to get Baggage.current(). It returns empty.

Hmm, when you call Span.current() in the same place what does it return?

Have you configured the javaagent to use custom propagators? (OTEL_PROPAGATORS/otel.propagators)
Can you share some example code that's causing you problems?

[vishalgoel1988]

Thanks for the info.

We tried setting jaeger propogators and as well default, both did not work .
We are getting below Span:

Span -> RecordEventsReadableSpan{traceId=84c1a283c159d840e8dfd82e1bd290af, spanId=6386b9491394fdc4, parentSpanContext=ImmutableSpanContext{traceId=84c1a283c159d840e8dfd82e1bd290af, spanId=46bacdcf7a47c199, traceFlags=01, traceState=ArrayBasedTraceState{entries=[]}, remote=false, valid=true}, name=classxxx.functionxxx, kind=INTERNAL, attributes=AttributesMap{data={thread.id=443, thread.name=Jetty-443, code.namespace=classxxx, code.function=functionxxx}, capacity=128, totalAddedValues=4}, status=ImmutableStatusData{statusCode=UNSET, description=}, totalRecordedEvents=0, totalRecordedLinks=0, startEpochNanos=1634052731578261438, endEpochNanos=0}
Code

@GET
@Consumes(MediaType.APPLICATION_JSON)
@Path("/{command}")
@Produces(MediaType.APPLICATION_JSON)
public Response fnctionxxx(
@PathParam("command") String command,
@QueryParam("param") String text)
{
Span.current();
Baggage.current();
}

Please check.

[jkwatson]

Just as a datapoint, I have verified that with v1.6.2, baggage does make it into Baggage.current() if you send in the w3c baggage header, and are using the propagation configuration defaults. This was using a toy app written using sparkjava.

[mateuszrzeszutek]

Hmm, what about the javaagent settings? Can you share your javaagent configuration? (well the OTEL_PROPAGATORS is the most important one in this case)

[vishalgoel1988]

We got the issue. We were passing header as baggage: value1 instead of baggage: key1=value1

Thanks all for the help.

[vishalgoel1988]

We got the issue. We were passing header as baggage: value1 instead of baggage: key1=value1