OTLP: errors 401 or 404 for the metrics endpoint

[mariomac]

When using the OTEL_EXPORTER_OTLP_HEADERS environment variable to authenticate the connection to send either metrics and traces to the OTEL endpoint, Beyla sends traces correctly but logs the following message:

2023/09/18 14:03:02 failed to upload metrics: failed to send metrics to https://otlp-gateway-prod-eu-west-0.grafana.net/otlp/v1/metrics: 401 Unauthorized

Also 404 errors have been observed in some versions of the OTEL endpoint.

The cause is that the underlying OpenTelemetry Go library is not accepting the OTEL_EXPORTER_OTLP_HEADERS value for metrics (it works for traces)

Workaround

You can explicitly set the OTEL_EXPORTER_OTLP_METRICS_HEADERS environment variable with the same value as OTEL_EXPORTER_OTLP_HEADERS so Beyla will send the proper headers to the metrics endpoint.

[mariomac]

Closing, as I can't reproduce it.

[gouthamve]

We just reproduced it and found the root-cause. Fix incoming.

[mariomac]

Fix: #446