setup OpenTelemetry with AWS Lambda and chain to parent traceId

[jimkont]

I set quarkus with opentelemetry and AWS lambda as described in the respective documentation pages, however, it seems that I am missing some parts of the process.

while the traces are correctly propagated to jaeger, they are not chained with the parent traceId that is passed in the lambda invocation. A similar node-based lambda function we have does that automatically so not sure if there is any extra configuration we need to perform in quarkus to pickup the traceId from the lambda invocation context.
Does anyone have any experience in this type of setup?

[quarkus-bot[bot]]

/cc @brunobat (opentelemetry,tracing), @matejvasek (amazon-lambda), @patriot1burke (amazon-lambda), @radcortez (opentelemetry,tracing)

[brunobat]

What propagation mechanism are you assuming? AWS default context propagation uses xray... Are you including that propagator on the Quarkus project?

[brunobat]

According to OTel, you need to include the X-Ray propagator.
Please check: https://opentelemetry.io/docs/reference/specification/trace/semantic_conventions/instrumentation/aws-lambda/#aws-x-ray-environment-span-link

[brunobat]

Also check the Quarkus documentation on how to add that propagation, here: https://quarkus.io/guides/opentelemetry#propagators

[jimkont]

update: in the lambda request that is sent by the central service, I see this header:

x-amz-client-context: eyJDdXN0b20iOnsidHJhY2VwYXJlbnQiOiIwMC1kZWQ4ZGUwMzNjYWRlMjY3MThkM2Y5ZjUwMzUwZDg0ZS0xMWU0OTllYTI4NWFkODIwLTAxIn19

which is a base64 encoding of {"Custom":{"traceparent":"00-ded8de033cade26718d3f9f50350d84e-11e499ea285ad820-01"}}

however, when I run the configuration locally by submitting the call to the dev endpoint (http://localhost:8080) as well as online, the AWS lambda Context.clientContext object is null, is there another way I can access the value of x-amz-client-context header here?

[brunobat]

Please check my last comments. You MUST use X-Ray propagation with AWS Lambdas and therefore include the related libs in the Quarkus service, otherwise, nothing will work.

[jimkont]

Thanks for the links, looking into it already.

However I need some clarification, the AWS lambda invocation docs states that X-Amz-Client-Context can be used to populate the context.clientObject with client data, however, I see that in the quarkus lambda library you ignore that header when you populate the context object and only read Lambda-Runtime-Client-Context. Is there another place that you read this information or does quarkus discard it completely?

[brunobat]

Because you need the X-Ray propagator included in

<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-extension-aws</artifactId>
</dependency>

For Quarkus to do that.

[jimkont]

Many apologies for persisting but I think I am confused.

From my understanding, io.opentelemetry:opentelemetry-extension-aws is set to read the X-Amzn-Trace-Id header, which is required for X-Ray. I have tried to use that library without success.

In my previous message, I was referring to the X-Amz-Client-Context header. This is a standard AWS Lambda header used to pass client metadata to the handler and this is how our central service is passing the tracing information to all lambda functions. From what I understand this header has nothing to do with OpenTelemetry.
The Quarkus AmazonLambdaContext class seems to be ignoring this header when building the handler context and I think this is where the issue in my case lies.

[jimkont]

created this issue as a follow-up #31676,
thanks for your help so far @brunobat