AWS instrumentation not adding request specific info (e.g. SQS message attributes)

[cfbao]

Component

OpenTelemetry.Instrumentation.AWS

Package Version

Package Name	Version
OpenTelemetry.Extensions.Hosting	1.9.0
OpenTelemetry.Instrumentation.AWS	1.1.0-beta.6

Runtime Version

net8.0

Description

AWS instrumentation used to add OTel context (e.g. traceparent) as message attributes in SQS & SNS messages in 1.1.0-beta.4 and lower versions.
However, since 1.1.0-beta.5, it's no longer sending these message attributes.

Steps to Reproduce

using Amazon.SQS;
using Amazon.SQS.Model;
using OpenTelemetry.Trace;

var builder = Host.CreateApplicationBuilder(args);
builder.Services.AddOpenTelemetry().WithTracing(builder => builder
	.AddAWSInstrumentation(o => o.SuppressDownstreamInstrumentation = true)
	.AddSource("SqsOtel"));
builder.Services
	.AddSingleton<IAmazonSQS, AmazonSQSClient>()
	.AddHostedService<Worker>()
var host = builder.Build();
host.Run();

class Worker(IAmazonSQS sqs) : BackgroundService
{
	private const string QueueUrl = "<my-queue-url>";

	protected override async Task ExecuteAsync(CancellationToken stoppingToken)
	{
		while (!stoppingToken.IsCancellationRequested)
		{
			var sendResponse = await sqs.SendMessageAsync(new()
			{
				QueueUrl = QueueUrl,
				MessageBody = "Hello from SQS!",
			}, stoppingToken);

			Console.WriteLine($"""
				sent:
					{sendResponse.MessageId}
				""");

			var receiveResponse = await sqs.ReceiveMessageAsync(new ReceiveMessageRequest
			{
				QueueUrl = QueueUrl,
				MaxNumberOfMessages = 10,
				WaitTimeSeconds = 20,
				MessageAttributeNames = ["All"],
			});

			foreach (var msg in receiveResponse.Messages)
			{
				Console.WriteLine($"""
					received:
						{msg.MessageId}
						{msg.MessageAttributes.Count}
						{msg.MessageAttributes.GetValueOrDefault("traceparent")?.StringValue}
					""");

				await sqs.DeleteMessageAsync(new()
				{
					QueueUrl = QueueUrl,
					ReceiptHandle = msg.ReceiptHandle,
				}, stoppingToken);
			}

			await Task.Delay(1000, stoppingToken);
		}
	}
}

Expected Result

We see a "traceparent" header in the message, with console output like this:

sent:
        771550da-5352-4c69-9c88-b086473bd5e8
received:
        771550da-5352-4c69-9c88-b086473bd5e8
        1
        00-0649b17db128b84151f3fb0b637e3e94-23abe4da559fa36c-01
sent:
        53684bc1-bfdb-4207-a402-fa3593ad4d83
received:
        53684bc1-bfdb-4207-a402-fa3593ad4d83
        1
        00-d820bbe16eb01fe39d6fefa159547d73-b0488a6848d530f1-01

(This is the case when using 1.1.0-beta.4)

Actual Result

We don't see any message attributes. Console output is like this:

sent:
        291e4e71-6baf-40dc-b909-409a213926b6
received:
        291e4e71-6baf-40dc-b909-409a213926b6
        0

sent:
        e4cb6cb6-9758-41c9-a19a-b9620befd034
received:
        e4cb6cb6-9758-41c9-a19a-b9620befd034
        0

sent:
        d8e5b4cd-a3c0-4a8d-8dcf-8d0fa7c9a2e9
received:
        d8e5b4cd-a3c0-4a8d-8dcf-8d0fa7c9a2e9
        0

Additional Context

This is likely caused by the change to not add an OTel handler before the Marshaller here:
https://github.com/open-telemetry/opentelemetry-dotnet-contrib/pull/1974/files#diff-6195c65008a9029c5e01df0537d42a4bada85ccf240e4e3e86cd4224e8b8a84aL37-L42

Now, all OTel related processing is done after the request has materialized, and any changes to IRequestContext.OriginalRequest won't affect the request body that's actually sent.

[muhammad-othman]

Hello @cfbao ,
This missing request specific info is fixed in the latest release.
Feel free to reach out if you faced any other issues.