Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update mapper from statsd_exporter #70

Closed
matthiasr opened this issue Dec 5, 2018 · 8 comments
Closed

Update mapper from statsd_exporter #70

matthiasr opened this issue Dec 5, 2018 · 8 comments

Comments

@matthiasr
Copy link
Contributor

to get prometheus/statsd_exporter#169 (fix for clobbering captures in unsuccessful branches of the FSM).

Blocked by #69.

@matthiasr
Copy link
Contributor Author

I'm afraid I spoke too soon, updating the vendoring causes more issues that I need to sort out first, so I filed this issue to keep track of it @pnyheim

@pnyheim
Copy link
Contributor

pnyheim commented Dec 6, 2018

OK. I'll keep an eye on it and await the release. In the meantime we will test using the glob_disable_ordering: true workaround.

@pnyheim
Copy link
Contributor

pnyheim commented Jan 24, 2019

Hi. @matthiasr
Any chance of getting this merged and released soon?

@matthiasr
Copy link
Contributor Author

unfortunately I haven't had time to upgrade the Prometheus client library myself yet, a PR for that would greatly help things along.

@pnyheim
Copy link
Contributor

pnyheim commented Jan 24, 2019

I'd be happy to help, but don't have much experience with go, so I cant make any promises wrt. upgrading the Prometheus client library.

But the reason I'm asking now is that we have discovered some anomalies in our metrics - that we suspect being related to the mappings in graphite_exporter.

What we are seeing is lots of (for lack of a better word) garbled metric names in prometheus.
Eg. metricnames mixed with other metricnames or labelvalues metric_name______some_label_value or metric_name_____some_other_partial_metric_name
And we are also seeing labelvalues that should not be there for some of our labels. Eg. we have a counter where a label result should have the values successful or failed - but also see labelvalues from another metrics.

This is only happening sporadically - and very hard to troubleshoot.

Parts of what I explain above looks like the issue you fixed in statsd_exporter - which is why we would like to test the fix in graphite_exporter without using the workaround from above.

We might also decide to downgrade to 0.2.0 where we did not see any such anomalies.

This is probably not the right place to ask for this, but it's kinda hard to explain so I'll just do that for now.
Maybe @fffonion can take a look at it as well - as you fixed the issue in the FSM - and I have a bad feeling that this could be a bug with it.

@pnyheim
Copy link
Contributor

pnyheim commented Feb 11, 2019

We had to downgrade graphite_exporter today because of these anomalies.
All "garbled" metrics disappeared the second we downgraded, so I feel fairly confident that this is a real bug in the graphite_exporter - and probably in the FSM that is also used in statsd_exporter.

image

It seems strange that we should be the only ones that experience this problem. Any chance you could have a look at it @matthiasr or @fffonion?

@raypettersen
Copy link

This is a very nasty bug. Is it on anyone's radar?

@matthiasr
Copy link
Contributor Author

I'm sorry I left this one around for so long. Work has been demanding, and I am still intimidated and confused by Go modules … a thousand thanks to @fffonion for picking this up!

I'm going to cut a release with the fix shortly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants