Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Filter duplicate duration metrics #208

Closed
Tracked by #198
Ishatt opened this issue Sep 26, 2023 · 1 comment · Fixed by #332
Closed
Tracked by #198

Filter duplicate duration metrics #208

Ishatt opened this issue Sep 26, 2023 · 1 comment · Fixed by #332
Assignees
@morgsmccauley

Comments

@Ishatt
Copy link
Contributor

Ishatt commented Sep 26, 2023

Description

The EXECUTION_DURATION is written only after the indexer succeeds, this means that when it is not running, or erroring, the last successful duration is exposed via the /metrics endpoint. Grafana will continue to scrape this metric giving the appearance that the given indexer is running at a constant duration.

We need to somehow filter out these duplicate metrics. Reseting the metric to 0 on error/skip doesn’t work as it will be reset almost instantly after a successful run, and because Grafana scrapes infrequently, it is likely that the actual successful metric will be missed.
@Ishatt Ishatt mentioned this issue Sep 26, 2023
Closed
@morgsmccauley
Copy link
Collaborator

We should use Histogram rather than Gauge

@morgsmccauley morgsmccauley linked a pull request Oct 26, 2023 that will close this issue
fix: Use Histogram to prevent stale metrics being scraped #332
Merged
@morgsmccauley morgsmccauley mentioned this issue Oct 26, 2023
Merged
morgsmccauley added a commit that referenced this issue Oct 26, 2023
@morgsmccauley
fix: Use Histogram to prevent stale metrics being scraped (#332)
0c9fa36 
This PR migrates the `EXECUTION_DURATION` to `Histogram` from `Gauge`.
With `Gauge`s, the previously recorded metric value will continue to be
scraped creating a flat-line graph in Grafana. Histograms allow us to
better approximate the metric over a specific time range, removing these
flat-lines.

Resolves #208
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@morgsmccauley morgsmccauley

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: Use Histogram to prevent stale metrics being scraped near/queryapi
2 participants
@morgsmccauley @Ishatt