Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

the same metric with different labels override each other #65

Closed
sciffer opened this issue Nov 19, 2015 · 14 comments
Closed

the same metric with different labels override each other #65

sciffer opened this issue Nov 19, 2015 · 14 comments

Comments

@sciffer
Copy link

sciffer commented Nov 19, 2015 •
edited
Loading

So, If I try to send to the same push gateway the same metric but with different values in the labels they override each other using post(and put) - it seems that it replaces the metric no matter if replace is set to false or true.
@beorn7
Copy link
Member

beorn7 commented Nov 21, 2015 •
edited by roidelapluie
Loading

That's by design.

If the grouping key and the name of the metric is the same, then metrics are always replaced, regardless if you use POST and PUT. POST will not replace metrics with a different __name__, though.

@beorn7 beorn7 closed this as completed Nov 21, 2015
@radut
Copy link

radut commented Mar 20, 2017

"If the grouping key and the name of the metric is the same..." What grouping key ?

Can I push same metric name different labels on one instance of pushgateway with different jobs,so that they don't override each other?

@sciffer
Copy link
Author

sciffer commented Mar 20, 2017 via email •
edited
Loading

@radut
Copy link

radut commented Mar 20, 2017

And what is the grouping key. What fields are in this grouping key?

@beorn7
Copy link
Member

beorn7 commented Mar 20, 2017

The README.md on https://github.com/prometheus/pushgateway explains the grouping key (just search for "grouping key" on that page to find all the occurrences).

In general, it makes more sense to ask questions like this on the prometheus-users mailinglist rather than in a GitHub issue.

@sreque
Copy link

sreque commented May 30, 2017 •
edited
Loading

This still feels like a bug or at least a design flaw to me. Say for instance, you have an ephemeral job like an AWS Lambda function processing files uploaded to an S3 bucket. There is no meaningful grouping key for this setup. It looks like you would have to stick a StatsD server in front of your push gateway in order to get meaningful service-level metrics.

After discussing with some other folks, I feel like a push gateway really only serves one use case: I can't scrape my endpoints due to my network architecture or other security concerns.

Even then, PGs do not appear to properly handle counters, and, by derivative, histograms or summaries at all. They only support gauges, and only with "last-write-wins" aggregation. If you want to generate metrics for ephemeral jobs, if possible, it seems like you would be much better off running an exporter daemon host on each machine running the jobs, and have the jobs push metrics updates to these exporters on each run. Then prometheus could scrape the job endpoints as normal.

The above limitation has an unfortunate consequence that you are forced to essentially have an "instance" dimension to all your metrics, even if this dimension isn't useful or meaningful to you, resulting in major and unnecessary timeseries bloat. I again can't see any way to work around this other than through using a separate aggregator like StatsD.

@brian-brazil
Copy link
Contributor

Something statsd-like is one of the appropriate ways to deal with Lambda-like things. The pushgateway does not do temporal aggregation.

@Sudhar287 Sudhar287 mentioned this issue Mar 5, 2020
Closed
@alexklibisz
Copy link

For others looking, you can work around the last-write-wins semantic by using your labels as the grouping key. That took a while to figure out, and I'm not 100% sure if there's any consequence to doing it this way. In my case I'm using the Java client. I used the push(CollectorRegistry registry, String job, Map<String, String> groupingKey) method and passed my <String, String> map of labels as the groupingKey.

@XiaoWeiKIN
Copy link

This is a bad design

@allenporter allenporter mentioned this issue Aug 30, 2021
Closed
@volodya-lombrozo volodya-lombrozo mentioned this issue Oct 4, 2021
Open
volodya-lombrozo added a commit to volodya-lombrozo/scout that referenced this issue Oct 4, 2021
@volodya-lombrozo
fix(#3): add applicationId as label during pushing metrics to pushgat…
8c63947 
…eway. Workaround prometheus/pushgateway#65 (comment)
@ElSamhaa
Copy link

ElSamhaa commented Jul 7, 2022

I'd hate to comment on an old issue, but I must say the current design is not good. The PUT/POST behaviors are counter-intuitive to say the least.

POST method should be associated with posting new/replacing.
PUT method should be associated with granular updating of the group (i.e metrics with the same name).

@mdebord-dlr
Copy link

If the grouping key and the name of the metric is the same, then metrics are always replaced

Even with unique labels? Does anyone know where this is documented, I haven't seen it mentioned and I'm facing this same issue.

The fact that unique labels don't actually make it unique is very counter intuitive.

@sarabala1979 sarabala1979 mentioned this issue Mar 30, 2023
Merged
@contrudar
Copy link

does anybody know how to work it around? For instance, how to use approach of @alexklibisz but using plain POST request

@sciffer
Copy link
Author

sciffer commented Oct 14, 2023 •
edited
Loading

I know it's an old thread that I opened up long time a go, but just in case someone need counter&histogram support for push gateway - you can try using the aggregation push gateway: https://github.com/zapier/prom-aggregation-gateway
While they solved the counter and histogram support, they broke the logic for gauges as those are treated as if they were counters (not sure why).

@tkporter tkporter mentioned this issue Mar 28, 2024
Merged
tkporter added a commit to hyperlane-xyz/hyperlane-monorepo that referenced this issue Apr 2, 2024
@tkporter
Different key funder job name (#3506)
9b06f3b 
### Description

- My changes in #3491 / #3492 were kind of accurate - according to
prometheus/pushgateway#65 even using pushAdd
will displace existing metrics. It's all very confusing!
- I tested this with a script that pushed some test metrics to the push
gateway - basically by having the key funder use a different job name
than the mainnet2 one, the metrics each of them push won't conflict with
one another

### Drive-by changes

<!--
Are there any minor or drive-by changes also included?
-->

### Related issues

<!--
- Fixes #[issue number here]
-->

### Backward compatibility

<!--
Are these changes backward compatible? Are there any infrastructure
implications, e.g. changes that would prohibit deploying older commits
using this infra tooling?

Yes/No
-->

### Testing

<!--
What kind of testing have these changes undergone?

None/Manual/Unit Tests
-->
yorhodes pushed a commit to hyperlane-xyz/hyperlane-monorepo that referenced this issue Apr 14, 2024
@tkporter @yorhodes
Different key funder job name (#3506)
6636ef7 
### Description

- My changes in #3491 / #3492 were kind of accurate - according to
prometheus/pushgateway#65 even using pushAdd
will displace existing metrics. It's all very confusing!
- I tested this with a script that pushed some test metrics to the push
gateway - basically by having the key funder use a different job name
than the mainnet2 one, the metrics each of them push won't conflict with
one another

### Drive-by changes

<!--
Are there any minor or drive-by changes also included?
-->

### Related issues

<!--
- Fixes #[issue number here]
-->

### Backward compatibility

<!--
Are these changes backward compatible? Are there any infrastructure
implications, e.g. changes that would prohibit deploying older commits
using this infra tooling?

Yes/No
-->

### Testing

<!--
What kind of testing have these changes undergone?

None/Manual/Unit Tests
-->
letonchanh pushed a commit to letonchanh/hyperlane-monorepo that referenced this issue Apr 25, 2024
@tkporter @letonchanh
Different key funder job name (hyperlane-xyz#3506)
b9acb78 
### Description

- My changes in hyperlane-xyz#3491 / hyperlane-xyz#3492 were kind of accurate - according to
prometheus/pushgateway#65 even using pushAdd
will displace existing metrics. It's all very confusing!
- I tested this with a script that pushed some test metrics to the push
gateway - basically by having the key funder use a different job name
than the mainnet2 one, the metrics each of them push won't conflict with
one another

### Drive-by changes

<!--
Are there any minor or drive-by changes also included?
-->

### Related issues

<!--
- Fixes #[issue number here]
-->

### Backward compatibility

<!--
Are these changes backward compatible? Are there any infrastructure
implications, e.g. changes that would prohibit deploying older commits
using this infra tooling?

Yes/No
-->

### Testing

<!--
What kind of testing have these changes undergone?

None/Manual/Unit Tests
-->
@bpoole6
Copy link

bpoole6 commented Aug 12, 2024

The gauge issue and having to restart prom-aggregation-gateway to delete stale metrics was a no go for our use case. That's why I created the Metrics Accumulator https://github.com/bpoole6/metrics-accumulator

The Metrics Accumulator doesn't add gauge values together, it instead takes the latest value pushed for a gauge (There is some customization here).

It also has a TTL for your metrics.

@GeorgeTsagk GeorgeTsagk mentioned this issue Aug 21, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests