Amazon SQS: Transactions and Distributed Tracing Headers for receiveMessage

[astorm]

The current SQS implementation does not automatically generate transactions or handle the distributed tracing traceparent/tracecontext headers for the receiveMessage API method. We currently recommend users use the startTransaction API to decide when/how they want transactions started for this use case.

We should consider implementing automatic handling of both the transaction creation and distributed tracing headers for SQS queues.

Problems to Solve

Where to set the traceparent header.

• SQS has no concept of a "header". The only place we have available to store a header is with the message attributes. This it not ideal, as these are the same attributes a user might use to set data on their messages and Amazon limits messages to 10 headers. In addition to taking up 10% of the available headers, it's also unclear what name we should use for this message. Since this is a space users can set their own headers an un-namespaced traceparent might not be a "safe" value (this is unlike HTTP where traceparent is considered safe since it's a W3C standard that's designed to be shared by multiple vendors.

When to Create a Transaction

• There's no clear/ideal place to start/stop a transaction. The receiveMessage method is a polling style API with a callback. Users call receiveMessage, passing in a callback. The SQS SDK will call this callback once it's finished fetching data and can return between 1 and 10 messages

  • A transaction could be automatically started for the duration of the callback. This approach might interfere if users want to have a transaction per message in order to use distributed tracing.
  • Another approach is to attempt to start a transaction per message -- however the response object isn't typed -- it's an object that includes a Messages property that contains an array of Message objects -- but the message objects are just native javascript objects. Wrapping either the iteration or access of these message is a non-trivial task.

• In addition to the challenges of where to set a distributing tracing header, there's also the question of how transaction parenting should work with this polling-via-callback style API. A single call to receiveMessage can return messages that may have started in different transactions. We don't have an implementation for anything like span links (which would allow a spans to have multiple parent-like spans), and attempting to link via a transaction for each message runs into the previously mention problem of not being able to wrap either the iteration or access to native message array.

[trentm]

https://github.com/elastic/apm/blob/main/specs/agents/tracing-instrumentation-messaging.md#message-receptionconsumption

SQS .receiveMessage() fits the "Active message reception" section and this part:

a span link SHOULD be added for each message with Trace Context. This includes the case where the size of the batch of received messages is one.

This will be handled via #2593. I'm closing this one as a duplicate.