fix store query bug when running loki in single binary mode with boltdb-shipper #2655
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## master #2655 +/- ##
==========================================
+ Coverage 61.22% 61.46% +0.24%
==========================================
Files 172 172
Lines 13360 13380 +20
==========================================
+ Hits 8179 8224 +45
+ Misses 4431 4410 -21
+ Partials 750 746 -4
|
lzh-lab
reviewed
Dec 6, 2021
Comment on lines
+177
to
+191
| // query ingester for whole duration. | ||
| if ingesterMLB == -1 { | ||
| i := &interval{ | ||
| start: queryStart, | ||
| end: queryEnd, | ||
| } | ||
|
|
||
| if limitQueryInterval { | ||
| // query only ingesters. | ||
| return i, nil | ||
| } | ||
|
|
||
| // query both stores and ingesters without limiting the query interval. | ||
| return i, i | ||
| } |
There was a problem hiding this comment.
Why query ingester for whole duration ？
There was a problem hiding this comment.
The logic here is inconsistent with the documentation.
There was a problem hiding this comment.
This is to support the rare usecase where you run Loki with multiple single binaries and each of them storing the data on local filesystem without any sharing.
lzh-lab
reviewed
Dec 6, 2021
Comment on lines
+203
to
+209
| if !limitQueryInterval { | ||
| i := &interval{ | ||
| start: queryStart, | ||
| end: queryEnd, | ||
| } | ||
| return i, i | ||
| } |
There was a problem hiding this comment.
The microservices deployment mode with object store will run here no matter if there is an overlap.
ingester:
query_store_max_look_back_period: 0
querier:
query_ingesters_within: 2h
This case will also query ingester for whole duration.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
When running loki with boltdb-shipper in single binary mode and doing queries not touching the recent data I found that store was not being queried. The problem appears to be here which considers end time of query instead of
time.Now()while calculatingadjustedEndof query. If you do a query for yesterday with 1h interval andIngesterQueryStoreMaxLookbackis set to2hthen theadjustedEndwould be set to older value than start time of query hence the condition here would be true and result in not querying the store.I have refactored the code to do all the calculations in a separate function which has following properties:
IngesterQueryStoreMaxLookbackwould take precedence overQueryIngestersWithinconfig.IngesterQueryStoreMaxLookbackwould do separate non overlapping queries to stores and ingesters whileQueryIngestersWithinwould do same queries to both which is same as how it was before.Checklist