Fixed --store.grpc.series-sample-limit #2858
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
3 tasks
bwplotka
approved these changes
Jul 8, 2020
| @@ -722,12 +721,16 @@ func blockSeries( | |||
| s.refs = append(s.refs, meta.Ref) | |||
| } | |||
| if len(s.chks) > 0 { | |||
| if err := chunksLimiter.Reserve(uint64(len(s.chks))); err != nil { | |||
There was a problem hiding this comment.
We could potentially limit chunk bytes instead TBH, but this is good for a start
There was a problem hiding this comment.
Or both. In Cortex we would need by count :)
There was a problem hiding this comment.
Opened an issue: #2861
| if reserved := atomic.AddUint64(&l.reserved, num); reserved > l.limit { | ||
| // We need to protect from the counter being incremented twice due to concurrency | ||
| // while calling Reserve(). | ||
| l.failedOnce.Do(l.failedCounter.Inc) |
|
Thanks @bwplotka for your feedback. I should have addressed the comments and fixed the doc. |
pracucci
force-pushed
the
fix-series-sample-limit
branch
from
745d1de
to
61d8981
Compare
bwplotka
approved these changes
Jul 10, 2020
|
Looks like my manual rebase through Github UI (there were conflicts) was not successful.. 😢 I tried ;p |
Signed-off-by: Marco Pracucci <[email protected]>
Signed-off-by: Marco Pracucci <[email protected]>
pracucci
force-pushed
the
fix-series-sample-limit
branch
from
8982200
to
bc3fcd2
Compare
|
Thanks! |
2 tasks
|
Hi @pracucci, And I'm wondering that do we have some metrics about loaded/fetched samples per-query? So we can determine a query is rejected by this series limit or other reason. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
In this PR I've changed the
--store.grpc.series-sample-limitimplementation to apply the limit to the sum of "samples" fetched across all queried blocks instead of individually apply the limit to each block.Few notes:
ChunksLimiterFactoryto not pass a static limit toNewBucketStorebecause I would like to use it in Cortex too and in Cortex we have dynamic limits (limits can be reloaded live, without restarting the process)Fixes #2845.
Verification
Existing + new unit tests.