RFC - Pipeline Component Telemetry

[djaglowski]

This PR adds a RFC for normalized telemetry across all pipeline components. See #11343

edit by @mx-psi:

• Announced on #otel-collector-dev on 2024-10-23: https://cloud-native.slack.com/archives/C07CCCMRXBK/p1729705290741179
• Announced on the Collector SIG meeting from 2024-10-30

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 91.56%. Comparing base (68f0264) to head (7925012).
Report is 100 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #11406      +/-   ##
==========================================
- Coverage   92.15%   91.56%   -0.60%     
==========================================
  Files         432      441       +9     
  Lines       20291    23856    +3565     
==========================================
+ Hits        18700    21844    +3144     
- Misses       1228     1640     +412     
- Partials      363      372       +9     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[codeboten]

Thanks for opening this as a RFC @djaglowski!

[djaglowski]

Based on some offline feedback, I've broadened the scope of the RFC, while simultaneously clarifying that it is intended to evolve as we identify additional standards.

[jaronoff97]

a few questions, I really like this proposal overall :)

[jpkrohling]

I think this needs normative language. Setting the otel.signal for the OTLP receiver for its bootstrap operations are certainly misleading to people operating the collector who are unaware of the inner working of this specific component (ie, that it's a singleton).

So:

attributes MUST be omitted if the corresponding component instances

[djaglowski]

This is a tricky topic and I'm not sure we can be so strict. It certainly makes sense to me that e.g. logs generated while initializing the singleton should not be attributed to one signal or pipeline. However, we can still attribute metrics to a particular signal (e.g. if otlp receiver emits 10 logs and 20 metrics, do you want a count of "30 items" or "10 logs" and "20 metrics". Maybe this is a good argument for splitting the proposed metrics by signal type, e.g. produced_metrics, produced_logs, etc. This would allow those metrics to share the same set of attributes with other signals produced by the instance.

[jpkrohling]

I know this states "pipeline component telemetry", and that extensions aren't technically part of a pipeline, but it feels wrong to leave them out: otel.component.kind and .id could definitely apply to them as well.

[djaglowski]

The scope of this effort has been increased a lot already. Can we leave extensions for another proposal? Personally I don't feel I have enough expertise with extensions to author such details.

[jpkrohling]

How costly? I remember talking to someone about this in the past, and they mentioned that it's not that expensive, given that it just delegates to what already exists in protobuf:

opentelemetry-collector/pdata/ptrace/pb.go

Line 22 in 9907ba5

return pb.Size()

It would be nice to have benchmarks to have data backing this (or the other) claim. I would definitely see as very useful to have a histogram of item/batch sizes and having it as optional means that people might only find out about it when they'd benefit from having historical data in the first place.

[djaglowski]

Suggested change

	These are high cost to compute, so by default they should be disabled (and not calculated).
	These may be high cost to compute, so by default they should be disabled (and not calculated). This default setting may change in the future if it is demonstrated that the cost is generally acceptable.

How's this wording?

[jpkrohling]

This is clearly a tracing case for me :-) The rule of thumb to me is: is the information related to a particular transaction? Then it should go into a span.

[djaglowski]

Makes sense if we agree that we should capture a span for the consume function.

[jpkrohling]

Context is passed down, isn't it? We can definitely instrument the ingress part of the component, and ask components to add span links if they are messing with the context. This way, the trace for a pipeline with a batch processor would end at the batching processor, but a new trace with a span link would be created pointing to the originating batch request.

[djaglowski]

I see, so instead of closing the span when data comes out, we close it when the consume func returns?

I think the duration of the span would be meaningful only for some synchronous processors, and could be meaningful for syncronous connectors (e.g. if they create and link spans to represent the work associated with the incoming data). But what about asynchronous components? Do we accept that the span is just measuring a quick handoff to the internal state of the component? Is this going to be misleading to users?

[jpkrohling]

Some of my comments might have been discussed before, in which case, feel free to ignore me and just mark the items as resolved.