Default labels on exported metrics for Prometheus

[lucasoares]

Hello.

I was using metrics SDK version 0.30.0 and I'm now updating my code with the 0.33.0 to keep up with latest pre-GA version.

When I was using the version 0.30.0 all the exported metrics to (Prometheus Exporter) had my resource labels from the OTEL_RESOURCE_ATTRIBUTES and OTEL_SERVICE_NAME environment variables plus SDK versions.

After the update these labels are no longer returned and a target_info metric is now exposed (without SDK labels) which I can disable with WithoutTargetInfo, but I can't manage to make my metrics returning default labels again...

Before:

foo_bar_housekeeper_task_latency_bucket{service_name="foo_bar",service_version="0.6.4",task="custom_metric",telemetry_sdk_language="go",telemetry_sdk_name="opentelemetry",telemetry_sdk_version="1.7.0",le="0"} 0

After:

foo_bar_housekeeper_task_latency_bucket{task="custom_metric",le="0"} 0
target_info{service_name="foo_bar",service_version="0.6.4"} 0

Thanks.

[MrAlias]

This is how OpenTelemetry as a whole has decided to export this data. What the latest version exports is compliant with the OpenTelemetry specification. The previous versions were not. Changing to the old format is not something we are able to do.

I'm pretty sure the idea is if you want you can use Prometheus rewrite rules to annotate metrics with the data as you want.

[lucasoares]

This is how OpenTelemetry as a whole has decided to export this data. What the latest version exports is compliant with the OpenTelemetry specification. The previous versions were not. Changing to the old format is not something we are able to do.

I'm pretty sure the idea is if you want you can use Prometheus rewrite rules to annotate metrics with the data as you want.

There is any documentation available on how to do it? To make prometheus export to always set these metrics?

[MrAlias]

I don't know of any. Someone with more familiarity of Prometheus would need to speak to this.

[malud]

Hi,
since we use this pkg @avenga/couper and have to ensure backward compatibility we came up to a Gatherer Wrapper.

This works with v0.33 and those labels are applied to system metrics too (go, process):
(maybe os.getenv(OTEL_*) by hand)

import (
  prom "github.com/prometheus/client_golang/prometheus"
  dto "github.com/prometheus/client_model/go"
  otelprom "go.opentelemetry.io/otel/exporters/prometheus"
)

func newPromExporter() (*otelprom.Exporter, *WrappedRegistry, error) {
	const yourService = "a-name"
	const yourServiceVersion = "v1.x.x"

	strPtr := func(s string) *string { return &s }

	registry := NewWrappedRegistry(prom.NewRegistry(), &dto.LabelPair{
		Name:  strPtr("service_name"),
		Value: strPtr(yourService),
	}, &dto.LabelPair{
		Name:  strPtr("service_version"),
		Value: strPtr(yourServiceVersion),
	})

	registry.MustRegister(collectors.NewGoCollector())
	registry.MustRegister(collectors.NewProcessCollector(
		collectors.ProcessCollectorOpts{
			Namespace: "xxx", // name prefix
		},
	))

	promExporter, err := otelprom.New(otelprom.WithRegisterer(registry))

	return promExporter, registry, err
}

// registry.go

import (
	prom "github.com/prometheus/client_golang/prometheus"
	dto "github.com/prometheus/client_model/go"
)

var (
	_ prom.Gatherer   = &WrappedRegistry{}
	_ prom.Registerer = &WrappedRegistry{}
)

type WrappedRegistry struct {
	labels       []*dto.LabelPair
	promRegistry *prom.Registry
}

func NewWrappedRegistry(promRegistry *prom.Registry, labels ...*dto.LabelPair) *WrappedRegistry {
	return &WrappedRegistry{
		labels:       labels,
		promRegistry: promRegistry,
	}
}

func (wr *WrappedRegistry) Gather() ([]*dto.MetricFamily, error) {
	families, err := wr.promRegistry.Gather()
	if err != nil {
		return nil, err
	}

	// working on ptrs
	for _, f := range families {
		for _, m := range f.Metric {
			m.Label = append(m.Label, wr.labels...)
		}
	}
	return families, nil
}

func (wr *WrappedRegistry) Register(collector prom.Collector) error {
	return wr.promRegistry.Register(collector)
}

func (wr *WrappedRegistry) MustRegister(collector ...prom.Collector) {
	wr.promRegistry.MustRegister(collector...)
}

func (wr *WrappedRegistry) Unregister(collector prom.Collector) bool {
	return wr.promRegistry.Unregister(collector)
}

related commit: coupergateway/couper@17ccc70

Hope that helps @lucasoares

[lucasoares]

Thanks, I will test it...

edit: I will use your solution for now. Thanks!

[dashpole]

Regarding #3405 (comment), the query to join a metric with target_info should look something like:

my_metric * on(instance, job) group_left(service_name, service_version, etc) target_info

It should be possible to use recording rules to do this at ingest time, if using a Prometheus server to scrape.

[dashpole]

The exporter currently follows the opentelemetry specification for prometheus exporters. If the current behavior needs to be changed, we should open an issue in the specification repo.

The solutions discussed above are probably the best workarounds for now.

I propose we close this issue.