[feat][misc] PIP-264: Copy OpenTelemetry resource attributes to Prometheus labels

[dragosvictor]

PIP-264

Motivation

The OpenTelemetry collector can be configured to expose the metrics as a Prometheus server, simplifying the deployment and migration of the telemetry pipeline. Pulsar supports this feature out of the box.

There is a catch though: the Prometheus exporter did not copy the OpenTelemetry resource attributes over to corresponding Prometheus labels, so vital information was lost in this process. This limitation is described in open-telemetry/opentelemetry-java#6108, which has since been fixed. The present PR leverages this improvement, instructing the OpenTelemetry SDK to migrate all resource attributes over to Prometheus as labels.

Modifications

• During OpenTelemetry SDK initialization in OpenTelemetryService, configure any Prometheus exporters to copy all resource attributes as labels. Caveat: At the time this happens, the Prometheus server is already initialized and listening. Reconfiguring it on-the-go is not possible, so the server is closed and recreated with the desired configuration.

Verifying this change

• Make sure that the change passes the CI checks.

This change modified tests and can be verified as follows:

• Updated integration test OpenTelemetrySanityTest#testOpenTelemetryMetricsPrometheusExport to validate the labels are attached to the metrics exposed by the Prometheus exporter.

Does this pull request potentially affect one of the following parts:

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics: As described above
• Anything that affects deployment

Documentation

• doc
• doc-required [feat][doc] PIP-264: Add OpenTelemetry managed cursor metrics reference pulsar-site#929
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository: dragosvictor#38