prometheus receiver handling of lone _created metrics

[seankhliao]

Component(s)

receiver/prometheus

What happened?

Description

Metrics from a prometheus (but not openmetrics) client library ending with _created don't have their values set.

Steps to Reproduce

Run go program:

package main

import (
	"net/http"

	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promauto"
	"github.com/prometheus/client_golang/prometheus/promhttp"
)

func main() {
	ctr := promauto.NewCounter(prometheus.CounterOpts{
		Name: "my_thing_created",
	})
	ctr.Add(300)
	http.ListenAndServe(":8080", promhttp.Handler())
}

Run collector (see below for config)

Expected Result

A metric with name my_thing_created and value 300.

Actual Result

a metric with name my_thing_created and value 0, with a wrong start timestamp.

2024-01-05T20:22:18.263Z	info	ResourceMetrics #0
Resource SchemaURL: 
Resource attributes:
     -> service.name: Str(app)
     -> service.instance.id: Str(127.0.0.1:8080)
     -> net.host.port: Str(8080)
     -> http.scheme: Str(http)
ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope otelcol/prometheusreceiver 0.91.0
Metric #0
Descriptor:
     -> Name: my_thing_created
     -> Description: 
     -> Unit: 
     -> DataType: Sum
     -> IsMonotonic: true
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
StartTimestamp: 1970-01-01 00:05:00 +0000 UTC
Timestamp: 2024-01-05 20:22:18.259 +0000 UTC
Value: 0.000000

Collector version

otelcol-contrib version 0.91.0

Environment information

OpenTelemetry Collector configuration

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: 'app'
          scrape_interval: 5s
          static_configs:
            - targets: ['127.0.0.1:8080']
  
processors:
  filter:
    metrics:
      metric:
        - name != "my_thing_created"

exporters:
  debug:
    verbosity: detailed

service:
  pipelines:
    metrics:
      receivers:
        - prometheus
      processors:
        - filter
      exporters:
        - debug

Log output

-

Additional context

The metric looks like this in the scrape output:

# HELP my_thing_created 
# TYPE my_thing_created counter
my_thing_created 300

OpenMetrics uses _created as one of the special suffixes for defining the start timestamp of a given time series: https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#suffixes

given the TYPE hint, the metric shouldn't be treated as a timestamp, since it's an exact match, not with an extra _created suffix appended.

It looks like the collector is treating the metric as a timestamp (300s == 5 min past unix epoch), but also for the wrong metric name, since it it were a timestamp, it should be for a metric named my_metric instead of my_metric_created

[github-actions]

Pinging code owners:

• receiver/prometheus: @Aneurysm9 @dashpole

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dashpole]

Thanks for the report. I'll look into it when I have time.

[jmichalek132]

So I took a look at this, and found the cause here:

opentelemetry-collector-contrib/receiver/prometheusreceiver/internal/metricfamily.go

Line 300 in 788f03e

if strings.HasSuffix(metricName, metricSuffixCreated) {

if metrics has the _created suffix, it's value isn't set, instead created gets set.
In theory we could do something like

if strings.HasSuffix(metricName, metricSuffixCreated) && (mf.metadata.Type == "timestamp" || mf.metadata.Type == "") {
    mg.created = v
} else {
    mg.value = v
}

but I am not sure if that's the best approach.
I'll raise this as a topic in the OpenTelemetry Prometheus Workgroup.

[dashpole]

I think we would need to ensure that the suffix is in addition to the normal metric name (that we looked up in the metadata cache).

[dashpole]

Openmetrics spec demonstrating created timestamp: https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#overall-structure

Discussed this at the SIG meeting. If the metricfamily name + _created == the series name, then we should use it as a created timestamp.

[jmichalek132]

This was fixed in #30808, could you please close this one @dashpole ?