Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix for prometheus backend panic'ing when using unclustered agents #288

Merged
merged 1 commit into from
Jul 3, 2024

Conversation

wolfeidau
Copy link
Contributor

@wolfeidau wolfeidau commented Jul 2, 2024

If a customer is running unclustered agents and using the prometheus backend then the CLI will panic when building the metrics.

panic: inconsistent label cardinality: expected 1 label values but got 0 in []string(nil)

goroutine 1 [running]:
github.com/prometheus/client_golang/prometheus.(*GaugeVec).WithLabelValues(...)
        github.com/prometheus/[email protected]/prometheus/gauge.go:238
github.com/buildkite/buildkite-agent-metrics/v5/backend.(*Prometheus).Collect(0x14000144618, 0x140008187c0)
        github.com/buildkite/buildkite-agent-metrics/v5/backend/prometheus.go:65 +0x8b0
main.main.func1()
        github.com/buildkite/buildkite-agent-metrics/v5/main.go:182 +0xc4
main.main()
        github.com/buildkite/buildkite-agent-metrics/v5/main.go:194 +0x1160

This change leaves the pruning of empty cluster names to prometheus, for example this is buildkite_total_idle_agent_count with an example of clustered and unclustered agents.

buildkite_total_idle_agent_count{instance="whatever:9999", job="prometheus"} 
buildkite_total_idle_agent_count{cluster="Default cluster", instance="whatever:9999", job="prometheus"}

Copy link
Contributor

@DrJosh9000 DrJosh9000 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@wolfeidau wolfeidau marked this pull request as ready for review July 2, 2024 05:43
@wolfeidau wolfeidau merged commit f1798ab into master Jul 3, 2024
1 check passed
@wolfeidau wolfeidau deleted the fix_prometheus_backend_for_unclustered_agents branch July 3, 2024 23:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants