AWS SDK v3 traces not showing

[haimprecise]

What version of OpenTelemetry are you using?

@opentelemetry/core: 1.12.0
@opentelemetry/api: 1.3.0
@opentelemetry/instrumentation-aws-sdk: 0.34.1

What version of Node are you using?

18.14.0-alpine

What did you do?

Updated from AWS SDK v2 to v3:

"@aws-sdk/client-s3": 3.321.1
"@aws-sdk/client-sns": 3.321.1
"@aws-sdk/client-sqs": 3.321.1

And deployed to our AKS v1.23

What did you expect to see?

Exported traces for AWS SQS messages

What did you see instead?

No traces at all

Additional context

When debugging, the patchV3 functions are not called at all.

[Flarna]

Are you using some bundler like esbuild/webpack (see here)?

Are you creating the AWS SDK instrumentation before the AWS SDK module is required?

Is your application using CJS or ESM modules?

[haimprecise]

Were are using CJS, but the OpenTelemtry is wrapped with a NestJS Module implementing its OnApplicationBootstrap, It is called once all modules have been initialized, but before listening for connections.

According to the opentelemetry-instrumentation docs, it seems like the mechanism won't kick in, am I right?

[Flarna]

Not an expert on NestJS but it sounds like thats to late. Most stable approach is to move OTel init into a dedicated file and preload it via -r command line argument.

[haimprecise]

Okay I will try call the instrumentation code before the app bootstrap.
I will share the results.

Thanks

[haimprecise]

I call the Opentelemtry code before my app in instantiated.

This is what I use:

this.otelExporter = new PrecisePrometheusExporter({
port: 8081,
});

  this.otelSDK = new NodeSDK({
    metricReader: this.otelExporter,
    spanProcessor: new BatchSpanProcessor(
      !!process.env.NODE_ENV &&
      process.env.NODE_ENV?.toUpperCase() !== "LOCAL"
        ? new JaegerExporter()
        : new ConsoleSpanExporter()
    ),
    contextManager: new AsyncLocalStorageContextManager(),
    textMapPropagator: new CompositePropagator({
      propagators: [
        new JaegerPropagator(),
        new W3CTraceContextPropagator(),
        new W3CBaggagePropagator(),
        new B3Propagator(),
        new B3Propagator({
          injectEncoding: B3InjectEncoding.MULTI_HEADER,
        }),
      ],
    }),
    instrumentations: [
      new HttpInstrumentation({
        ignoreIncomingPaths: ["/health", "/metrics", /^\/api.*$/],
        ignoreOutgoingUrls: [/^.*amazonaws\.com.*$/, /^.*unleash.*$/],
      }),
      new GrpcInstrumentation(),
      new RedisInstrumentation(),
      new ExpressInstrumentation(),
      new DnsInstrumentation(),
      new AwsInstrumentation({
        suppressInternalInstrumentation: true,
      }),
      new PrismaInstrumentation(),
      new NestInstrumentation(),
      new PinoInstrumentation(),
    ],
  });
  
  this.otelSDK.start();

I can verify with debugging that the instrumentation code is triggered before the AWS imports.
However, the patch functions are not called.

Here is my tsconfig.json:

  "compilerOptions": {
    "module": "commonjs",
    "declaration": true,
    "removeComments": true,
    "emitDecoratorMetadata": true,
    "experimentalDecorators": true,
    "allowSyntheticDefaultImports": true,
    "target": "es2020",
    "sourceMap": true,
    "outDir": "./dist",
    "baseUrl": "./",
    "incremental": true,
    "allowJs": true,
    "esModuleInterop": true,
    "moduleResolution": "node"
  }
}```

and I'm also excluding the `node_modules` in the `tsconfig.build.json`.

Any ideas why it's not working?

[Flarna]

fyi @carolabadeer @blumamir - seems like instrumentation of AWS SDK has some issue here

[carolabadeer]

Can you try adding the AWSXRayPropagator to the list of propagators?

[haimprecise]

I've added them but no change.

The issue seems at the level of patching via middleware-stack, non of the patchV3 functions are triggered, although I start the NodeSDK prior to my app initialization.

[dyladan]

@carolabadeer I'm going to assign this to you since you're the code owner. Can you please categorize this as P1-P4 and I'll add the appropriate label.

[haimprecise]

Hey @carolabadeer,
Any updates? Can I assist?

[carolabadeer]

Hi @haimprecise, from what I learned, the prometheus exporter only exports metrics and not traces. This is probably why you are seeing this behaviour. Are you able to use another exporter instead (like the OTLP exporter)?

[carolabadeer]

@dyladan can we add a p4 label to this issue?

[dyladan]

@carolabadeer it looks like they're also using the Jaeger exporter for traces.

[haimprecise]

Indeed, we are using the JaegerExporter.

[carolabadeer]

@haimprecise can you provide a simple reproducible code snippet I can use to reproduce the issue on my end? Which receiver and exporter are you using?

[dyladan]

Looks like NewRelic had the same issue and they fixed it here: newrelic/node-newrelic-aws-sdk#194

We may be able to apply a similar fix

[johnkoehn]

We are seeing this as well. It is limiting our ability to have end to end OTEL traces. What can I do to help get this across the finish line @carolabadeer?

[carolabadeer]

I noticed this issue, unlike #1596, mentions aws-sdk packages v3.321.1. @johnkoehn are you using v3.321.1 or v3.363.0?

[justinasfour04]

Do we have a timeline on a fix? @carolabadeer can you increase the priority please

[carolabadeer]

I am working with @blumamir on a fix. In the meantime as a workaround, please use @AWS-SDK < v3.363.0

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days.

[ithompson-gp]

Seeing this issue on the latest aws-sdk instrumentation lib (0.36.0)
Node 18
Any updates on a fix? Or else we will have to roll back to aws-sdk v2

[trentm]

IIUC #1626 was meant to have resolved this. I am closing this issue.

Since then, there was another breakage in aws-sdk v3 instrumentation, fixed in #1913.

Because there have been multiple issues and version releases of both aws-sdk packages and OTel packages, I think a new issue should be opened if anyone encounters an issue with aws-sdk instrumentation, so we can be clear on the versions involved.

[chagasjoao]

Still not working for:

"@opentelemetry/instrumentation-aws-sdk": "^0.40.0",
"@aws-sdk/client-dynamodb": "^3.556.0",
"@aws-sdk/lib-dynamodb": "^3.556.0",