Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can TTL'd metrics be resurrected if pushed into the exporter again later? #353

Closed
mgalgs opened this issue Dec 17, 2020 · 2 comments
Closed
Labels

Comments

@mgalgs
Copy link

mgalgs commented Dec 17, 2020

We've had issues with stale metrics hanging around (#129) and wanted to start using TTLs to get rid of them (#164).

My question is about metrics that are rarely updated. Based on my understanding of #164, if a metric passes its TTL without being pushed into the statsd exporter it will be marked as stale and will no longer be reported to Prometheus. So what happens if the metric is reported again sometime later? Will it be resurrected and reported to Prometheus again or is it permanently quieted after hitting its TTL? This is all on a per "label values set" basis, of course.

It looks like before #246 any TTL'd metrics would have stayed dead forever, but that PR seems to have fixed that. I just want to confirm my understanding before using the option.

Thanks!

@matthiasr
Copy link
Contributor

Metrics that have expired are forgotten and removed; if you start sending them again later they will reappear. Counters, summaries, gauges will be reset in the process. This means you will have gaps in the rarely sent metrics, and counters may not work the way you expect them to at all (you cannot rate() over a single sample).

If possible, the best way to avoid stale metric issues is to tie the exporter lifecycle to the application, for example by deploying it as a sidecar.

I'm going to close the issue since there is nothing for us to do here, if you do encounter weirdness around TTL handling please do file a new issue!

@mgalgs
Copy link
Author

mgalgs commented Dec 18, 2020

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants