Add downsampling metrics #1244
Merged
Add downsampling metrics #1244
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mattrco
commented
Jun 11, 2019
bwplotka
approved these changes
Jun 11, 2019
|
Thanks for reviewing, have updated to use |
bwplotka
approved these changes
Jun 12, 2019
| } | ||
| } | ||
|
|
||
| type DownsampleMetrics struct { |
There was a problem hiding this comment.
I think it can be private, right?
GiedriusS
reviewed
Jun 12, 2019
| @@ -65,6 +92,8 @@ func runDownsample( | |||
| } | |||
| }() | |||
|
|
|||
| metrics := newDownsampleMetrics(reg) | |||
There was a problem hiding this comment.
Could we use extprom.WrapRegistererWithPrefix("thanos_compact_downsample", reg) here?
There was a problem hiding this comment.
I think it makes sense but with thanos_compact prefix, but not a blocker.
| func newDownsampleMetrics(reg *prometheus.Registry) *DownsampleMetrics { | ||
| m := new(DownsampleMetrics) | ||
|
|
||
| m.downsamples = prometheus.NewCounterVec(prometheus.CounterOpts{ |
There was a problem hiding this comment.
Then we would only have total here
|
Merging in this form, we can improve that later once we actually need to reuse this component. |
|
Thanks @mattrco ! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When
compactgoes into its downsampling step, no app metrics are reported, which makes our dashboard look a bit bare for several hours (sometimes days if we have a backlog). Adding a couple of metrics means we can alert if no compactions or downsamples have happened in x period.This is a little less clean than I'd like due to the two callers of
downsampleBuckethaving slightly different contexts, however I think it's a reasonable compromise. Let me know what you think!The metrics are labelled in the same way as compact metrics for ease of comparison.
Changes
Add
thanos_compact_downsample_totalandthanos_compact_downsample_failures_totaldownsampling metrics.Verification
Tests are passing, will run with real workloads.