Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

serviceruntime metric errors #19

Closed
bytheway opened this issue Feb 8, 2018 · 4 comments · Fixed by #23
Closed

serviceruntime metric errors #19

bytheway opened this issue Feb 8, 2018 · 4 comments · Fixed by #23
Labels

Comments

@bytheway
Copy link

bytheway commented Feb 8, 2018

I think I'm running into a similar bug as #9, getting a 500 error for metrics from serviceruntime.googleapis.com:

4 error(s) occurred:

* collected metric stackdriver_consumed_api_serviceruntime_googleapis_com_api_request_count label:<name:"grpc_status_code" value:"0" > label:<name:"location" value:"us-central1" > label:<name:"method" value:"google.api.servicecontrol.v1.ServiceController.Check" > label:<name:"project_id" value:"______" > label:<name:"protocol" value:"" > label:<name:"response_code" value:"200" > label:<name:"response_code_class" value:"2xx" > label:<name:"service" value:"servicecontrol.googleapis.com" > label:<name:"unit" value:"1" > label:<name:"version" value:"v1" > gauge:<value:2 >  has label dimensions inconsistent with previously collected metrics in the same metric family...

Thanks for the great project!

@bytheway
Copy link
Author

bytheway commented Feb 8, 2018

Investigating further, it looks like google includes the credential_id label on most metrics, but it is missing on others.

@bytheway
Copy link
Author

bytheway commented Feb 8, 2018

(Sorry for the comment spam), This would also be helped by getting the go client fixed, as reported: prometheus/client_golang#355

@frodenas
Copy link
Member

Interesting, according to the docs, the consumed_api resource type should return the credential_id label.

Inconsistent label dimensions have been a problem for proxy exporters like this. An easy workaround will be to always add that label when querying for serviceruntime metrics, but 1) I don't like applying specific logic for particular metrics 2) it will only solve the problem for this particular metric, and we cannot guarantee other metrics will face the same problem in the future. Adding a generic helper for missing labels seems the way to go, let's keep an eye on prometheus/client_golang#355.

@frodenas frodenas added the bug label Feb 19, 2018
@bytheway
Copy link
Author

I think what is happening is that the consumed_api and the produced_api metrics are all mixed together in the same serviceruntime space. Here is what the produced apis are returning: https://cloud.google.com/monitoring/api/resources#tag_api

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
2 participants