Exporter metrics (otelcol_exporter_queue_capacity, otelcol_exporter_queue_size) are generated only for a single exporter on otel collector 0.96.0

[tqi-raurora]

Component(s)

No response

What happened?

Description

On version 0.96.0, when setting 2 different exporters, then querying localhost:8888/metrics, only one exporter is shown on the metrics otelcol_exporter_queue_size and otelcol_exporter_queue_capacity.
The exporter that is shown appears to be random, and can change if the service is restarted.
On version 0.92.0 this was working as expected, with metrics for both exporters being shown.

Steps to Reproduce

Create a configuration with 2 exporters (example: 2 otlp exporters)
Query local telemetry endpoint at localhost:8888/metrics

Expected Result

Both endpoints should be shown.
Example:

tqi_raurora@ni-26829-4p:~$ otelcol-contrib --version
otelcol-contrib version 0.92.0
tqi_raurora@ni-26829-4p:~$ curl -s 0:8888/metrics | grep ^otelcol_exporter | grep queue
otelcol_exporter_queue_capacity{exporter="otlp/jaeger-1",service_instance_id="721c4da6-9e52-4119-99e5-148e88e1dba6",service_name="otelcol-contrib",service_version="0.92.0"} 1000
otelcol_exporter_queue_capacity{exporter="otlp/jaeger-2",service_instance_id="721c4da6-9e52-4119-99e5-148e88e1dba6",service_name="otelcol-contrib",service_version="0.92.0"} 1000
otelcol_exporter_queue_size{exporter="otlp/jaeger-1",service_instance_id="721c4da6-9e52-4119-99e5-148e88e1dba6",service_name="otelcol-contrib",service_version="0.92.0"} 0
otelcol_exporter_queue_size{exporter="otlp/jaeger-2",service_instance_id="721c4da6-9e52-4119-99e5-148e88e1dba6",service_name="otelcol-contrib",service_version="0.92.0"} 0

Actual Result

Only metrics for one of the exporter is shown:

tqi_raurora@ni-26829-4p:~$ otelcol-contrib --version
otelcol-contrib version 0.96.0
tqi_raurora@ni-26829-4p:~$ curl -s 0:8888/metrics | grep ^otelcol_exporter | grep queue
otelcol_exporter_queue_capacity{exporter="otlp/jaeger-1",service_instance_id="c9c9dd6a-a367-42d1-a7c7-633c47b51a0a",service_name="otelcol-contrib",service_version="0.96.0"} 1000
otelcol_exporter_queue_size{exporter="otlp/jaeger-1",service_instance_id="c9c9dd6a-a367-42d1-a7c7-633c47b51a0a",service_name="otelcol-contrib",service_version="0.96.0"} 0

Collector version

0.96.0

Environment information

No response

OpenTelemetry Collector configuration

receivers:
  otlp/receiver:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318


exporters:
  otlp/jaeger-1:
    endpoint: 0.0.0.0:14317
    tls:
      insecure: true
  otlp/jaeger-2:
    endpoint: 0.0.0.0:14317
    tls:
      insecure: true


service:
  pipelines:
      
    traces:
      receivers: [otlp/receiver]
      processors: []
      exporters: [otlp/jaeger-1, otlp/jaeger-2]

Log output

No response

Additional context

I had opened the original issue on the contrib repository: open-telemetry/opentelemetry-collector-contrib#31645
But @crobert-1 determined it's an issue on the core repository, so I'm opening it here

[TylerHelmuth]

@tqi-raurora are both exporters taking traffic at the same time?

[tqi-raurora]

@TylerHelmuth
Yes
In my example I have a single otlp receiver, sending data to two otlp exporters.

But from what I tested it happens whenever theres more than one exporter (on our environment we got spans being sent to jaeger and metrics being sent to load balancer exporter, wich is a problem because we end up only with metrics for otlp exporter or load balancer exporter)

It seems whatever exporter gets to expose the metric is random, here I restarted the service without changing config.yaml and the exporter changed from otlp/jaeger-1 to otlp/jaeger-2:

tqi_raurora@ni-26829-4p:~$ curl -s 0:8888/metrics | grep ^otelcol_exporter | grep queue
otelcol_exporter_queue_capacity{exporter="otlp/jaeger-1",service_instance_id="e42887e0-4df6-4f7f-8ec6-9f5207636adc",service_name="otelcol-contrib",service_version="0.96.0"} 1000
otelcol_exporter_queue_size{exporter="otlp/jaeger-1",service_instance_id="e42887e0-4df6-4f7f-8ec6-9f5207636adc",service_name="otelcol-contrib",service_version="0.96.0"} 0

tqi_raurora@ni-26829-4p:~$ systemctl restart otelcol-contrib.service 

tqi_raurora@ni-26829-4p:~$ curl -s 0:8888/metrics | grep ^otelcol_exporter | grep queue
otelcol_exporter_queue_capacity{exporter="otlp/jaeger-2",service_instance_id="c2fb111a-e929-4350-82af-bc4049f8f6f2",service_name="otelcol-contrib",service_version="0.96.0"} 1000
otelcol_exporter_queue_size{exporter="otlp/jaeger-2",service_instance_id="c2fb111a-e929-4350-82af-bc4049f8f6f2",service_name="otelcol-contrib",service_version="0.96.0"} 0

[TylerHelmuth]

@dmitryax could this be related to any recent exporterhelper work?

[alexchowle]

Also seeing this issue on v0.99.0

[alexchowle]

This is quite a painful issue where you have a Collector that has multiple Exporters defined. I have to resort to the Collector's logs to determine which/how many of the Exporters are affected:

Sep 17 19:12:56 <HOSTNAME> <PROCESS>: 2024-09-17T18:12:56.346Z error exporterhelper/common.go:292 Exporting failed. Rejecting data. {"kind": "exporter", "data_type": "traces", "name": <EXPORTER_NAME>, "error": "sending queue is full", "rejected_items": 2}

Is there a potential due date for the associated milestone?

[alexchowle]

Looks like this was repaired in v0.105.0

[alexchowle]

Original (and repaired in 0.105.0) issue: #10444.

Extract from release notes:

exporterhelper: Fix incorrect deduplication of otelcol_exporter_queue_size and otelcol_exporter_queue_capacity metrics if multiple exporters are used. (#10444)

[tqi-raurora]

It's now working on the version I'm using, contrib v0.114.0:

Closing this issue