Added additional check for lazy Postings. #627
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bwplotka
force-pushed
the
store-profile
branch
from
34766cf
to
ca11572
Compare
|
The general idea is to merge this for more information in error cases, so PTAL (: |
|
So what I found out is that I stopped having those issues once I scaled the VMs running the store instance. This could indeed point to a race condition somewhere where CPU/Memory and IO would be saturated. I have kept a store instance with low resources to repro this issue even with your changes. I'll post the logs later. |
|
Thanks! |
|
@adrien-f Any progress? (: |
Debugging #335. Potentially broken preloadPosting is missing some postings, because of bad partitioning. Unit tests are there, but we might not get all edge cases? If with this check we will have still panic - it can only mean races. However, -race is not showing anything interesting on, even extended tests. Signed-off-by: Bartek Plotka <[email protected]>
bwplotka
force-pushed
the
store-profile
branch
from
ca11572
to
22cfc17
Compare
domgreen
approved these changes
Dec 12, 2018
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Debugging #335.
Potentially broken preloadPosting is missing some postings, because
of bad partitioning. Unit tests are there, but we might not get all edge cases?
If with this check we will have still panic - it can only mean races.
However, -race is not showing anything interesting on, even extended tests.
Would be nice to build testing tag with this and allow users to deploy it and test on setups that can repro store gateway panics.
PTAL @adrien-f
Signed-off-by: Bartek Plotka [email protected]