Why is my lambda showing a non recording span in the responseHook?

[Danwakeem]

• This only affects the JavaScript OpenTelemetry library
• This may affect other libraries, but I would like to get opinions here first

I am using the @opentelemetry/instrumentation-aws-lambda library inside of my lambda function. The lambda function I am looking at is has about 200 or so invocations a minute.

For some reason in the responseHook I am seeing the span printed out as a NonRecordingSpan and I am not sure why?

Even with this basic configuration I am displaying below in the AWS_LAMBDA_EXEC_WRAPPER I am seeing this happen. Does anyone have any ideas? 🤔

const { NodeTracerProvider } = require('@opentelemetry/sdk-trace-node');
const {
  InMemorySpanExporter,
  SimpleSpanProcessor,
} = require('@opentelemetry/sdk-trace-base');
const { AwsLambdaInstrumentation } = require('@opentelemetry/instrumentation-aws-lambda');
const { registerInstrumentations } = require('@opentelemetry/instrumentation');

const tracerProvider = new NodeTracerProvider();
const memoryExporter = new InMemorySpanExporter();
tracerProvider.addSpanProcessor(new SimpleSpanProcessor(memoryExporter));
tracerProvider.register();

registerInstrumentations({
  tracerProvider,
  instrumentations: [
    new AwsLambdaInstrumentation({
        responseHook: async (span) => {
           await tracerProvider.forceFlush();
           console.log('Wrapper span from event: ', span);
        }
    })
  ],
});

Also kind more interesting context, this appears to work on much lower invoked lambda functions. I have a few other test apps that I dropped this into where they are only invoked twice every thirty minutes and this works fine 🤷

[Danwakeem]

I figured out what my issue was 👇

I had to set a custom sampler to always sample the data. I am sure I could have messed with this a bit to have it not sample every request but fortunately I wanted to collected every execution 😎

So my to tie this back into the original question here is what the basic code looks like 👇

const { NodeTracerProvider } = require('@opentelemetry/sdk-trace-node');
const {
  InMemorySpanExporter,
  SimpleSpanProcessor,
} = require('@opentelemetry/sdk-trace-base');
const { AwsLambdaInstrumentation } = require('@opentelemetry/instrumentation-aws-lambda');
const { registerInstrumentations } = require('@opentelemetry/instrumentation');
const { SamplingDecision } = require("@opentelemetry/api");

const tracerProvider = new NodeTracerProvider({
  resource,
  sampler: {
    shouldSample(context, traceId, spanName, spanKind, attributes, links) {
      return {
        decision: SamplingDecision.RECORD_AND_SAMPLED,
        attributes,
      };
    },
    toString() {
      return 'babies-first-sampler';
    }
  }
});
const memoryExporter = new InMemorySpanExporter();
tracerProvider.addSpanProcessor(new SimpleSpanProcessor(memoryExporter));
tracerProvider.register();

registerInstrumentations({
  tracerProvider,
  instrumentations: [
    new AwsLambdaInstrumentation({
        responseHook: async (span) => {
           await tracerProvider.forceFlush();
           console.log('Wrapper span from event: ', span);
        }
    })
  ],
});

I still am a bit confused why the original implementation was not sending any spans for higher volume lambda functions but I am guessing it has something to do with how the sampling is determined when running in an ephemeral environment 🤷

[blumamir]

try to set disableAwsContextPropagation to true.
AWS has a bug in api gateway where they inject no sampling when x-ray is disabled, effectively disabling tracing for all downstream components.
you can read more here: #546

[Danwakeem]

@blumamir that also seems to work! Thank you for the reply 😎