no opentracing context value found for span context key x_b3_traceid for request [request_id]

[stackempty]

Hi,

I'm trying out opentracing with nginx-ingress-controller.

After enabling B3 using the following environment variables, the logs are filled with messages like below and the b3 headers are not being propagated.

environment vars

controller:
  extraEnvs:
  - name: DD_TRACE_PROPAGATION_STYLE_EXTRACT
    value: Datadog B3
  - name: DD_PROPAGATION_STYLE_INJECT
    value: Datadog B3
  - name: DD_AGENT_HOST
    valueFrom:
      fieldRef:
        fieldPath: status.hostIP
  - name: DD_ENV
    valueFrom:
      fieldRef:
        fieldPath: metadata.labels['tags.datadoghq.com/env']
  - name: DD_SERVICE
    valueFrom:
      fieldRef:
        fieldPath: metadata.labels['tags.datadoghq.com/service']

controller vars

controller:
  config:
    datadog-collector-host: "$DD_AGENT_HOST"
    datadog-service-name: "$DD_SERVICE"
    datadog-environment: "$DD_ENV"
    enable-opentracing: "true"
    location-snippet: |
      opentracing_trace_locations off;
      opentracing_tag resource.name $uri;
      opentracing_operation_name "$request_method $uri";
      opentracing_propagate_context;

log messages

2023/01/26 11:58:32 [error] 36#36: *10185 no opentracing context value found for span context key x_b3_traceid for request 00007F60CF41E3A0, client: [client_ip_addr], server: [domain].com, request: "POST /path HTTP/1.1", host: "[host].com", referrer: "http://localhost:3000/"
2023/01/26 11:58:32 [error] 36#36: *10185 no opentracing context value found for span context key x_b3_spanid for request 00007F60CF41E3A0, client: [client_ip_addr], server: [domain].com, request: "POST /path HTTP/1.1", host: "[host].com", referrer: "http://localhost:3000/"
2023/01/26 11:58:32 [error] 36#36: *10185 no opentracing context value found for span context key x_b3_sampled for request 00007F60CF41E3A0, client: [client_ip_addr], server: [domain].com, request: "POST /path HTTP/1.1", host: "[host].com", referrer: "http://localhost:3000/"

helm chart version: helm-chart-4.4.2

Could you please help/point me in the right direction?

Thank you

[dgoffredo]

I had a theory about why this might be happening, but it's wrong.

It would be helpful to have the incoming request headers for a request that produces the error log message.

Now my best guess at what's happening here is the following:

• A request comes in, and tracing is enabled.
• Either tracing context is extracted from request headers, or instead tracing context is created anew.
• Nginx finds an upstream and begins proxying the request.
• To inject the tracing context into the upstream request, the nginx-opentracing module inside of the Ingress Controller had previously added the following directives to the nginx location:
  • proxy_set_header x-b3-traceid $opentracing_context_x_b3_traceid;
  • and so on for the other B3 and Datadog propagation headers.
• The handler inside nginx for variables of the form $opentracing_context_<header with underscores> tells the tracing library (this is where Datadog comes in) to serialize its context to an in-memory mapping, as if it were serializing propagation headers. Nginx then looks up the header by name in this map.
• Nginx does not find the header in this in-memory map, and so prints an error to the log.

The question is, "why didn't nginx find the header in the in-memory map?" The Datadog tracer would have serialized all of the B3 headers, because you configured B3 as part of DD_PROPAGATION_STYLE_INJECT.

I could attempt a reproduction if I had more information about the incoming request.

For more immediate attention, consider contacting Datadog technical support: https://www.datadoghq.com/support/

[stackempty]

@dgoffredo Thanks for the response.

The message is logged for pretty much every request. Even a simple GET / to anything that's proxied.

[webratz]

Did any of you ever find a solution to this?

[dgoffredo]

I attempted a reproduction but was unable to get nginx to produce that error message.

[webratz]

I have reproduced that error message, but i could not get nginx to inject the b3 headers with any combination of the settings.

[dgoffredo]

The issue might be that the Datadog tracing library used by ingress-nginx, dd-opentracing-cpp, does not understand the DD_TRACE_PROPAGATION_STYLE_EXTRACT environment variable. It understands only DD_PROPAGATION_STYLE_EXTRACT and DD_PROPAGATION_STYLE_INJECT (note the lack of "TRACE").

Even with that, I can't get B3 propagation to work. I get Datadog headers only, and no error logging. Based on my reading of the code, propagation styles configured this way would not work at all, and yet you're getting an error message indicating that B3 was configured.

I thought that maybe the problem was that worker processes were not receiving the environment variables due to a lack of the following in the generated nginx.conf file's main section:

env DD_PROPAGATION_STYLE_INJECT;
env DD_PROPAGATION_STYLE_EXTRACT;

So, I added a relevant main-snippet to the ingress controller's ConfigMap. This did not change anything.

apiVersion: v1
data:
  allow-snippet-annotations: "true"
  datadog-collector-host: $HOST_IP
  enable-opentracing: "true"
  main-snippet: "env DD_PROPAGATION_STYLE_INJECT; env DD_PROPAGATION_STYLE_EXTRACT;"
kind: ConfigMap
# ...

Try the name change (no "TRACE") and the main-snippet mentioned above, and see if that changes the behavior of your program.

With my setup, the ingress controller silently omits any mention of B3.

[dgoffredo]

Update on the above: B3 propagation actually is working with my setup. I was hitting the wrong endpoint, so the nginx ingress controller was not even handling requests. Now I'm giving curl the right endpoint, and I do not see the error message reported.

[webratz]

Awesome, with these changes it seems to actually emit the b3 traces. nice!

[dgoffredo]

I've been thinking about this more, and I think that I know what's going on. I haven't tested it thoroughly, though, so take it with a grain of salt.

Feel free to ignore if you got it working and just want to move on. :)

The nginx-opentracing module (and nginx-datadog, for that matter) injects trace propagation headers by adding invisible proxy_set_header directives to traced locations. This has some undesirable side effects, but works. The proxy_set_header expands to the name of the header, and then the value is determined by an "nginx variable" of the form $opentracing_context_some_header_name, where "some_header_name" indicates the HTTP request header "some-header-name". The handler for this nginx variable lazily loads a map of trace context produced by the underlying tracer implementation.

The underlying tracer implementation is configured in the worker process, while the nginx configuration containing the invisible proxy_set_header happens in the master process, before forking the worker processes.

So, if you set DD_PROPAGATION_STYLE_INJECT in the ingress controller Deployment, that results in a set of invisible proxy_set_header directives that involve all of the styles mentioned in the environment variable. Then, nginx forks its worker processes and doesn't forward the environment variable. So then at runtime, when a request is being processed, nginx-opentracing asks the tracer, "please give me the value of x-b3-traceid." The tracer responds, "I don't have a value for that propagation header, because my injection style defaulted to 'Datadog'."

Modifying the nginx.conf to contain env DD_PROPAGATION_STYLE_INJECT; means that the tracer in the worker processes will be configured with the desired injection styles, and so will have populated the x-b3-traceid header, which nginx will then retrieve when honoring the relevant invisible proxy_set_header directive.

It sucks, but it's how it currently works.

We're evaluating how Datadog will integrate with the ingress controller in the future. I'll be sure that this issue does not apply in the new solution.