[receiver/Prometheus] inconsistent timestamps on metric points error

[venkateshsredhat]

Component(s)

receiver/prometheus

What happened?

Description

OpenTelemetryCollector pod shows below error :

2024-04-03T06:55:00.063Z warn internal/transaction.go:149 failed to add datapoint {"kind": "receiver", "name": "prometheus", "data_type": "metrics", "error": "inconsistent timestamps on metric points for metric prober_probe_total", "metric_name": "prober_probe_total", "labels": "{name="prober_probe_total", _source="", env="int", instance="", job="prometheus-self"}"}

otel collecotr config：

receivers: prometheus: config: global: external_labels: _source: "**example**" scrape_configs: - job_name: prometheus-self scrape_interval: 1m scrape_timeout: 10s metrics_path: /federate scheme: http honor_labels: false enable_http2: true kubernetes_sd_configs: - role: service namespaces: own_namespace: false names: - **example** selectors: - role: service label: "***example***" params: 'match[]': - '{__name__="prober_probe_total"}'

Is this a known bug ? Currently this warning is just present all over the log even though the metric gets exported fine but we would like to solve this issue .

Expected Result

No error warnings in the logs .

Actual Result

internal/transaction.go:149 failed to add datapoint {"kind": "receiver", "name": "prometheus", "data_type": "metrics", "error": "inconsistent timestamps on metric points for metric prober_probe_total", "metric_name": "prober_probe_total", "labels": "{name="prober_probe_total", _source="", env="int", instance="", job="prometheus-self"}"}

Collector version

v0.93.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

receivers:
        prometheus:
          config:
            global:
              external_labels:
                _source: "**example**"
            scrape_configs:
              - job_name: prometheus-self
                scrape_interval: 1m
                scrape_timeout: 10s
                metrics_path: /federate
                scheme: http
                honor_labels: false
                enable_http2: true
                kubernetes_sd_configs:
                  - role: service
                    namespaces:
                      own_namespace: false
                      names:
                        - **example**
                    selectors:
                      - role: service
                        label: "***example***"
                params:
                  'match[]':
                    - '{__name__="prober_probe_total"}'

Log output

No response

Additional context

No response

[github-actions]

Pinging code owners for receiver/prometheus: @Aneurysm9 @dashpole. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[crobert-1]

Potentially related: #22096

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• receiver/prometheus: @Aneurysm9 @dashpole

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dashpole]

I haven't seen it before. Can you share more about your setup? Is prober_probe_total from the kubelet? Are you scraping another Prometheus server?

[dashpole]

Is the issue happening for any other metrics?

[imrajdas]

Hi Team, I am also facing the same issue

2024-06-05T14:39:06.885Z	warn	internal/transaction.go:149	failed to add datapoint	{"kind": "receiver", "name": "prometheus", "data_type": "metrics", "error": "inconsistent timestamps on metric points for metric container_cpu_usage_seconds_total", "metric_name": "container_cpu_usage_seconds_total", "labels": "{__name__=\"container_cpu_usage_seconds_total\", cpu=\"total\", env=\"production\", id=\"/system.slice/containerd.service\", instance=\"iro-agent:8080\", job=\"opentelemetry-collector\"}"}
2024-06-05T14:39:06.885Z	warn	internal/transaction.go:149	failed to add datapoint	{"kind": "receiver", "name": "prometheus", "data_type": "metrics", "error": "inconsistent timestamps on metric points for metric container_cpu_usage_seconds_total", "metric_name": "container_cpu_usage_seconds_total", "labels": "{__name__=\"container_cpu_usage_seconds_total\", cpu=\"total\", env=\"production\", id=\"/system.slice/kubelet.service\", instance=\"iro-agent:8080\", job=\"opentelemetry-collector\"}"}
2024-06-05T14:39:06.886Z	warn	internal/transaction.go:149	failed to add datapoint	{"kind": "receiver", "name": "prometheus", "data_type": "metrics", "error": "inconsistent timestamps on metric points for metric container_memory_failures_total", "metric_name": "container_memory_failures_total", "labels": "{__name__=\"container_memory_failures_total\", env=\"production\", failure_type=\"pgfault\", id=\"/\", instance=\"iro-agent:8080\", job=\"opentelemetry-collector\", scope=\"container\"}"}
2024-06-05T14:39:06.886Z	warn	internal/transaction.go:149	failed to add datapoint	{"kind": "receiver", "name": "prometheus", "data_type": "metrics", "error": "inconsistent timestamps on metric points for metric container_memory_failures_total", "metric_name": "container_memory_failures_total", "labels": "{__name__=\"container_memory_failures_total\", env=\"production\", failure_type=\"pgfault\", id=\"/\", instance=\"iro-agent:8080\", job=\"opentelemetry-collector\", scope=\"hierarchy\"}"}

Note: I am fetching the CAdvisor metrics from kubelet API

[dashpole]

What collector versions are you using?

[dashpole]

@imrajdas can you share your prometheus receiver config?

[dashpole]

This might be an issue with metrics that set an explicit timestamp in the exposition.

[imrajdas]

Collector version- otel/opentelemetry-collector-contrib:0.96.0 (I have used the latest one also- 0.102.0, still the same issue)

exporters:
  prometheusremotewrite:
    external_labels:
      tenant: 78566289-1450-4a0c-b9ca-26a419058d86
    endpoint: <THANOS_RECEIVER>
extensions:
  health_check:
    endpoint: ${env:MY_POD_IP}:13133
receivers:
  prometheus:
    config:
      scrape_configs:
      - job_name: opentelemetry-collector
        scrape_interval: 10s
        static_configs:
        - targets:
          - 'agent:8080'
      - job_name: 'kube-api-blackbox'
        metrics_path: /probe
        params:
          module: [http_2xx]
        static_configs:
        - targets:
            - https://www.google.com
            - http://www.example.com
            - https://prometheus.io
        - targets:
            - http://paymentservice.bofa.svc.cluster.local:50051
            - http://frontend.bofa.svc.cluster.local:80
            - http://test.bofa.svc.cluster.local:80
          labels:
            service: 'internal-services'
        relabel_configs:
        - source_labels: [__address__]
          target_label: __param_target
        - source_labels: [__param_target]
          target_label: instance
        - target_label: __address__
          replacement: blackbox-exporter.imrajdaschi.svc.cluster.local:9115

service:
  extensions:
  - health_check
  pipelines:
    metrics:
      exporters:
      - prometheusremotewrite
      receivers:
      - prometheus

[imrajdas]

I have built a custom exporter called the Kubelet API to get the CAdvisor metrics and expose them.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• receiver/prometheus: @Aneurysm9 @dashpole

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[flenoir]

Hi, having same issue. Does anyone had solved this to get rid of error message ?

[flenoir]

After having a discussion in slack . it pointed me to the right direction. As explained in message, a "container_id" label was dropped. In my case, i also found that an "id" label as dropped. As i removed this labeldrop, the issue disappeared.