Limit for prometheus.googleapis.com

[ppippi-dev]

I use gcm_exporter but i meet some error

the log is here

caller=export.go:827 level=error component=gcm_exporter msg="send batch" size=200 err="rpc error: code = ResourceExhausted desc = Your metric descriptor quota has been exhausted: Limit for prometheus.googleapis.com/ is 25000."

can you help me? Please explain which quota should be changed

[lyanco]

Hey there,

This limit is on the number of unique metric names you can have. You should really never hit this in practice unless you're doing something like sending statsd-format metrics to us (with dimensionality encoded in the metric name instead of using labels) or putting timestamps in the metric name.

You can see what you're doing wrong by going to Metrics Explorer in Cloud Monitoring, unchecking the "show only active metrics" box in the metrics selector, and looking at your metrics under Prometheus Target. Likely culprits are exporters such as Vault, Spark, and Istio.

Once you've identified the malformed metrics (it should be obvious which have dimensionality encoded in the name), you can delete them by modifying the script here to find your metrics and delete them: #325 (comment)

Hope this helps.

[ppippi-dev]

Thanks your comment! I will check it. 😄