Warning: ADOT Collector v0.31.0 breaking change

[bryan-aguilar]

The ADOT team is aware of a breaking change made in v0.76.3 of the upstream OpenTelemetry collector-contrib package. This breaking change affects users of the prometheus and prometheusremotewrite exporter. Starting in version v0.76.3 the pkg.translator.prometheus.NormalizeName feature gate is enabled by default. The enabling of this feature gate results in the exporter using full prometheus metric name normalization rather than simple normalization. To read more about metric name normalization please refer to the normalization package documentation here.

The enabling of this feature gate could result in a change to the metric names exported by Collector metric pipelines using the prometheus receiver,prometheusremotewrite exporter, or prometheus exporter.

The ADOT team has made the decision to return the default state of this feature gate to disabled in ADOT Collector minor versions v0.29.x and v0.30.x. This will set the prometheus and prometheusremotewrite exporter to use simple metric name normalization by default.

Full metric name normalization can still be enabled by providing the --feature-gates=+pkg.translator.prometheus.NormalizeName flag at startup.

We highly advise that testing is done to determine if you are affected by this change before ADOT Collector release v0.31.0 is released.

Please feel free to ask any questions!

[bryan-aguilar]

The ADOT team is actively investigating on whether or not this should be a configurable behavior in the listed components. Please see open-telemetry/opentelemetry-collector-contrib#21743 issue upstream for updates.

[rapphil]

To enable the feature flag in ec2 instances, you have to update the /opt/aws/aws-otel-collector/etc/.env manually to add the --feature-gates=+pkg.translator.prometheus.NormalizeName

The content should look like

config="--config /opt/aws/aws-otel-collector/etc/config.yaml --feature-gates=+pkg.translator.prometheus.NormalizeName" 

To enable the feature flag in EC2 and ECS, you can pass the --feature-gates=+pkg.translator.prometheus.NormalizeName as a container argument/command line parameter.

[bryan-aguilar]

The upstream opentelemetry-collector-contrib has reverted this feature gate back to alpha status which disables it by default. Users can still opt into this feature gate if desired but no breaking changes will be occuring in v0.31.0 of the ADOT Collector.

I will be reverting the feature gates in the ADOT Collector distrubtion and remove the warnings from the readme. After those are merged that will close the issue.

The ADOT Team will continue to work upstream to align on prometheus name normalization.