New counter metric showing number of sampled/not sampled spans

[ArthurSens]

Component(s)

processor/tailsampling

Is your feature request related to a problem? Please describe.

I was debugging the input/output rate of spans of a few collectors and was super confused about the numbers I was seeing.

The metric rate(otelcol_receiver_accepted_spans[5m]) was showing around 12k spans/second, and rate(otelcol_processor_tail_sampling_count_traces_sampled{policy="mypolicy", sampled="false"}[5m]) was showing around 1k spans/second. Without any failures in the whole pipeline, I was expecting to see an output rate of 11k spans/second when looking at rate(otelcol_exporter_sent_spans[5m]), however, the numbers were showing only 6k spans/second.

@matej-g helped me with this and showed how the metric otelcol_processor_tail_sampling_count_traces_sampled is actually about traces, not spans, as the name suggests.

Describe the solution you'd like

It would be useful to have a metric showing the number of sampled/not sampled spans, to verify if input/output rate of spans is working as expected.

Describe alternatives you've considered

No response

Additional context

No response

[github-actions]

Pinging code owners:

• processor/tailsampling: @jpkrohling

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[ArthurSens]

Same thing for the metric otelcol_processor_tail_sampling_sampling_trace_dropped_too_early, if this metric increases I have no way of knowing how many spans were dropped

[jpkrohling]

I agree -- the only concern is about current users: what's the performance impact of adding this new counter? Can we wrap it in feature flags, so that users complaining about it can turn them off? If we hear no complaints for a couple of releases, we can remove the feature flag.