Bug or feature in golang client: xxx already contains grouping label yyy

[ascheman]

(Sorry, @beorn7, I already asked you for help in Discourse, and didn't see that we could also discuss my question here).

I run fluent-bit with a plugin to export metrics to Prometheus. I found the following issue when running multiple instances of it: https://github.com/neiman-marcus/fluent-bit-out-prometheus-metrics/issues/4

Tried to solve this problem by explicitly adding a Grouping instruction here like

@@ -620,7 +621,8 @@ func FLBPluginInit(plugin unsafe.Pointer) int {
        }
 
        // Initialize new metric with push gateway
-       pCtx.Pusher = push.New(pCtx.URL, pCtx.Job).Gatherer(registry)
+       pCtx.Pusher = push.New(pCtx.URL, pCtx.Job).Grouping("instance", pCtx.ConstantLabels["instance"]).Gatherer(registry)

In this case I get errors from the library:

level=error plugin=prometheus_metrics caller=out.go:789 id=indexed_counter msg="Could not put push to pushgateway, resetting retry counter, declaring failure data will be lost" err="pushed metric fluentbit_indexed_record_total (label:<name:\"instance\" value:\"7de1c1fe4f33\" > label:<name:\"method\" value:\"GET\" > label:<name:\"status_code\" value:\"200\" > counter:<value:25 > ) already contains grouping label instance"

I have drilled down the problem to the following line in Prometheus golang client library: https://github.com/prometheus/client_golang/blob/main/prometheus/push/push.go#L255

Is this a bug or a feature? I would think that the label should be allowed in both the grouping path as well as in the payload. The only test which makes sense to me is to check for equality of the value instead of the label name?

[beorn7]

Yeah, too many discussion forums for Prometheus… (told everyone so… ;)

The behavior is deliberate. At least it is documented as such: https://pkg.go.dev/github.com/prometheus/client_golang/prometheus/push#Pusher.Grouping

The basic idea is that grouping labels are akin to target labels and therefore should not collide with the metric’s own labels. I guess if the label value is the same, there is not really any harm done, it’s just a bit weird.

Can you perhaps just not set that label on the metric itself?

[ascheman]

Thanks for you reply, @beorn7, I now have a better understanding how it should work. Based upon this I could prepare a solution to my original problem and raised a respective pull-request: https://github.com/neiman-marcus/fluent-bit-out-prometheus-metrics/pull/5