Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[receiver/prometheusreceiver] Fix retrieval of aggregated metrics with no job/instance label #33565

Conversation

bacherfl
Copy link
Contributor

@bacherfl bacherfl commented Jun 14, 2024

Description: This PR fixes the retrieval of metrics where either the job or instance label is missing, and honor_labels is set to true. This can be the case for aggregated metrics coming from a federate endpoint. This PR introduces a fallback to using the job/instance labels from the scrape config for such metrics.

Link to tracking Issue: Fixes #32555

Testing:

  • Added a Unit test
  • Verified using the following config:
receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: 'federate'
          scrape_interval: 10s
          honor_labels: true
          params:
            'match[]':
              - '{__name__="cluster:node_cpu:sum_rate5m"}'
          metrics_path: '/federate'
          static_configs:
            - targets:
                - "localhost:9090"
exporters:
  debug:
    verbosity: detailed
  otlphttp:
    endpoint: ${env:OTLP_ENDPOINT}

  pipelines:
    metrics:
      receivers: [otlp,prometheus]
      exporters: [otlphttp, debug]

This was tested on a kind K8s cluster running the prometheus operator, with a port forward for the prometheus-k8s service created by the prometheus operator (therefore the localhost:9090 address in the target).

Signed-off-by: Florian Bacher <[email protected]>
@bacherfl bacherfl marked this pull request as ready for review June 17, 2024 10:53
@bacherfl bacherfl requested a review from a team June 17, 2024 10:53
@dashpole dashpole added the ready to merge Code review completed; ready to merge by maintainers label Jun 17, 2024
@evan-bradley evan-bradley merged commit 744141a into open-telemetry:main Jun 18, 2024
158 of 161 checks passed
@github-actions github-actions bot added this to the next release milestone Jun 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ready to merge Code review completed; ready to merge by maintainers receiver/prometheus Prometheus receiver
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Prometheus receiver fails on federate endpoint when job and instance labels are missing
4 participants