Instrumentation Library customization

[pellared]

The OTel Go community often asks to add customization to OTel Go Contrib instrumentation libraries such us (but not limited to):

1. Add custom "static" attributes to spans/measurements (so that the added attributes are always the same for all spans/measurements)
2. Add "dynamic" attributes to spans/measurements (so that the added attributes are different for each span/measurement)
3. Configure span options (like setting span kind, adding links, setting span name)
4. Add custom instrumentation scope attributes

Let's look at https://pkg.go.dev/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp as an example where such customization possibilities have been added via

1. https://pkg.go.dev/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp#Labeler
2. https://pkg.go.dev/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp#WithMetricAttributesFn
3. https://pkg.go.dev/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp#WithSpanOptions
4. https://pkg.go.dev/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp#WithSpanNameFormatter

The customizations that we added are seen like a precedence and the community often asks to add similar capabilities to other OTel Go Contrib instrumentation libraries (issues found in 3 minutes):

• Labeler support for traced spans opentelemetry-go-contrib#4231
• otelgrpc: add WithSpanOptions behavior to stats_handler opentelemetry-go-contrib#5430
• otelgrpc: Allow adding custom attributes to a request's metrics based on its payload opentelemetry-go-contrib#6026
• Support Custom Metric Attributes Per Request opentelemetry-go-contrib#6281
• Support WithSpanNameFormatter to otelecho opentelemetry-go-contrib#6363
• todo: adhere to the HTTP span naming semconv processing checklist opentelemetry-go-contrib#6375

However, I am not sure if such capabilities should be added as

1. The user can e.g. change the span kind using or create attributes which are not aligned with the semantic conventions and then others may blame that the instrumentation library does not follow semantic conventions.
2. Adding such customization in each instrumentation library is a huge maintenance burden. Shouldn't such things be handled by SDK using e.g. processors to solve such issues in a central place?

Do other languages have similar requests?
Do other languages also have such customization options?
Do other languages have similar problems?

CC @open-telemetry/go-maintainers

[dmathieu]

Some of those asks can be managed by folks themselves with custom handlers, since attributes can be added to spans at any moment of their duration. And spans are expected to high cardinality so that's a good thing.

I'm rather skeptical at the ask to add dynamic attributes to metrics though, as even though some providers allow reducing the cardinality of those metrics on their platform, allowing that provides a rather big gun for folks to shoot them in the foot.

[pellared]

I think this may be the way to add attributes to metrics measurements without polluting the instrumentation libraries:

• Support measurement processors in Metrics SDK #4298

CC @Blinkuu @reyang

[yurishkuro]

I would like to be able to have metrics produced automatically by collector's modules like confighttp to have a proper endpoint name as a label, instead of GET/POST.

[flc1125]

My suggestion is to write based on something like *http.Request in otelhttp.

Then, all other components should implement dependencies based on this otelhttp foundation, including but not limited to semconvutil.

[reyang]

I've seen similar asks in .NET, I think the key factors are:

• Access to the contextual data
• Efficiency

For example, folks who use ASP.NET Core instrumentation would want to manipulate the telemetry based on contextual information (async context, http context, thread context, etc.), the maintainers ended up providing a callback which allows the user to do whatever they want https://github.com/open-telemetry/opentelemetry-dotnet-contrib/tree/main/src/OpenTelemetry.Instrumentation.AspNetCore#enrich.

Another thing to consider - processors will see all the telemetry data for that particular provider, there is efficiency problem:

graph LR

InstrumentationLibraryA --> ProcessorX
InstrumentationLibraryB --> ProcessorX
InstrumentationLibraryC --> ProcessorX
ProcessorX --> ProcessorY --> ProcessorZ

Loading

If we want to manipulate the data from InstrumentationLibraryA, doing that in InstrumentationLibraryA would be more efficient than doing that in ProcessorX.

[pellared]

folks who use ASP.NET Core instrumentation would want to manipulate the telemetry based on contextual information

I think this is the key feature that cannot be easily done with processors.

the maintainers ended up providing a callback which allows the user to do whatever they want https://github.com/open-telemetry/opentelemetry-dotnet-contrib/tree/main/src/OpenTelemetry.Instrumentation.AspNetCore#enrich.

What about enriching metric measurements based on contextual information?

If we want to manipulate the data from InstrumentationLibraryA, doing that in InstrumentationLibraryA would be more efficient than doing that in ProcessorX.

This is a little questionable (the perf improvement will be very small) as a processor can simply make a conditional logic depending on the instrumentation scope (just a few CPU cycles to make a conditional jump). However, I think that more contextual information (e.g. access to HTTP request object) is the deal-breaker.

[cijothomas]

What about enriching metric measurements based on contextual information?

It is provided by some libraries natively. Eg: Asp.Net Core, HttpClient. It's not general purpose, but very specific to enrich metric emitted by those libraries. Its not very well documented, but here's a PR attempting to document it:
https://github.com/open-telemetry/opentelemetry-dotnet-contrib/pull/2059/files

[breedx-splk]

I don't understand where this is coming from or why these things are a challenge.

Add custom "static" attributes to spans/measurements (so that the added attributes are always the same for all spans/measurements)

This is kinda ridiculous, and I'm not sure why someone would want this over a resource attribute or whatever....but you can already do this trivially with a SpanProcessor. Where's the issue?

Add "dynamic" attributes to spans/measurements (so that the added attributes are different for each span/measurement)

You can already do this today with a SpanProcessor. Where's the issue?

if(span.getKind() == SpanKind.CLIENT) span.setAttribute("clientCount", myCount++);

Configure span options (like setting span kind, adding links, setting span name)

There are spec'd rules about span kind and span names, so I don't think they should be readily alterable after creation. Why would we want to allow two spans for the same piece of instrumentation to have different kinds or names depending on configuration!? Sounds maddening to me. Consistency wins here.

Span links area easily added with a SpanProcessor.

Add custom instrumentation scope attributes

That happens at a layer that's "higher" (above?) the instrumentation itself, right? Isn't it the responsibility of the thing creating/installing the instrumentation to do this? I don't think the instrumentation should be able to change its own scope (attributes).

[cijothomas]

The user can e.g. change the span kind using or create attributes which are not aligned with the semantic conventions

This is okay right? OTel instrumentations produce semantic convention compliant telemetry by default, but user can always override it to produce something else.
(Though I am unsure what would schema_url would point to then..)

[pellared]

TBH, I am not sure if it is okay (thus I created this discussion). My thoughts are exactly like @breedx-splk posted here: #4311 (comment)

The benefits of adding customizations like that to instrumentation libraries are that it gives more freedom and decouples from the SDK. People using different API implementations could still benefit from these customizations.

On the other side, I think that intention of the the SDK processors design was address these kind of issues and do not have to do these kind of things in all instrumentation libraries.

[pellared]

I have forgotten to say that still currently

I think that more contextual information (e.g. access to HTTP request object) is the deal-breaker.

[lmolkova]

Linking some relevant discussions:

• Clarify how Application Developers are supposed to set the Span status when using Instrumentation Libraries #4131 (comment)
• Propose Context-scoped attributes. oteps#207

Update:

• customization hooks for HTTP on spans in python https://github.com/open-telemetry/opentelemetry-python-contrib/blob/b6541f0bada9a0569a858337e2df27d5f6f3032a/instrumentation/opentelemetry-instrumentation-requests/src/opentelemetry/instrumentation/requests/__init__.py#L147

[s0]

I came across this issue when trying to add some additional attributes across all our metrics in some of my own services, and found it quite surprising that it wasn't supported by default in the Node.js SDK.

The approach I ultimately ended up taking was to require that any additional custom attributes were defined in the resource, and then allowing the exporters to be extended with functionality that would copy over specific attributes from the resources as specified, see: open-telemetry/opentelemetry-js#5253

const openTelemetrySdk = new NodeSDK({
  resource: new resources.Resource({
    [ATTR_SERVICE_NAME]: "test-otel-app",
    "deployment.environment.name": "production",
  }),
  metricReader: new PeriodicExportingMetricReader({
    exporter: extendExporterWithResourceAttributes(
      new ConsoleMetricExporter(),
      new Set(["deployment.environment.name", "process.pid"])
    ),
  }),
  instrumentations: getNodeAutoInstrumentations(),
});