Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

High cardinality caused by the url label #36

Open
prymitive opened this issue Aug 8, 2019 · 1 comment
Open

High cardinality caused by the url label #36

prymitive opened this issue Aug 8, 2019 · 1 comment

Comments

@prymitive
Copy link

url label contains the url of every request that was ever made against gin webserver. This can cause very high memory usage for prometheus server. What's worse this is out of control for the operator of a service running go-gin-prometheus and someone sending a high number of requests with random url can easily cause prometheus server scraping that webserver to run out of memory. This is a common misuse of prometheus metrics and there's even a tool written to detect and mitigate the most serious cases of cardinality explosion - https://blog.freshtracks.io/bomb-squad-automatic-detection-and-suppression-of-prometheus-cardinality-explosions-62ca8e02fa32

This is very easy to reproduce with curl:

for i in {1..1000} ; do curl -so /dev/null "localhost:8080/xxx$i" ; done```

Resulting in:

[...]
gin_requests_total{code="404",handler="main.notFound",host="localhost:8080",method="GET",url="/xxx100"} 1
gin_requests_total{code="404",handler="main.notFound",host="localhost:8080",method="GET",url="/xxx10"} 1
gin_requests_total{code="404",handler="main.notFound",host="localhost:8080",method="GET",url="/xxx1"} 1
gin_requests_total{code="404",handler="main.notFound",host="localhost:8080",method="GET",url="/xxx11"} 1
gin_requests_total{code="404",handler="main.notFound",host="localhost:8080",method="GET",url="/xxx12"} 1
gin_requests_total{code="404",handler="main.notFound",host="localhost:8080",method="GET",url="/xxx13"} 1
[...]

This was already reported in #34 and partially fixed in #35 but those changes are not enough.
All metrics already have the handler label that points to the request handler method that the request was routed to, this is enough to identify requests for any debugging / performance tracking.

Please remove the url label from metrics.

@satwell
Copy link

satwell commented Dec 6, 2019

Another option might be to use c.FullPath() rather than c.Request.URL.Path for the url label. FullPath was recently added in gin v1.5.0 and returns the path from the matched route or empty string for not found. So cardinality should be limited to the number of routes you have.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants