Speedup snapshot stale indices delete #613
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
❌ DCO Check Failed c3de5944883f79cd98fca7358bb43e7b30f7c083 |
|
✅ Gradle Wrapper Validation success c3de5944883f79cd98fca7358bb43e7b30f7c083 |
|
✅ Gradle Precommit success c3de5944883f79cd98fca7358bb43e7b30f7c083 |
nknize
requested changes
Apr 26, 2021
Fix for issue: elastic/elasticsearch#61513 Signed-off-by: Piyush Daftary <[email protected]>
|
✅ DCO Check Passed edc4db1 |
|
✅ Gradle Wrapper Validation success edc4db1 |
|
✅ Gradle Precommit success edc4db1 |
|
start gradle check |
nknize
added
enhancement
nknize
added
the
pending backport
|
Merged.. Thank you @piyushdaftary /cc @AmiStrn |
nknize
pushed a commit
to nknize/OpenSearch
that referenced
this pull request
Apr 27, 2021
Instead of snapshot delete of stale indices being a single threaded operation this commit makes it a multithreaded operation and delete multiple stale indices in parallel using SNAPSHOT threadpool's workers. Signed-off-by: Piyush Daftary <[email protected]>
nknize
added a commit
that referenced
this pull request
Apr 28, 2021
Instead of snapshot delete of stale indices being a single threaded operation this commit makes it a multithreaded operation and delete multiple stale indices in parallel using SNAPSHOT threadpool's workers. Signed-off-by: Piyush Daftary <[email protected]>
nknize
removed
the
pending backport
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fix for issue: elastic/elasticsearch#61513
Description
Current implementation cleanupStaleIndices() of snapshot deletion is very slow . Snapshot deletion code deletes each stale indices from repository one after another sequentially .
With this code changes instead of making snapshot delete of stale indices a single threaded operation we make it multithreaded operation and delete multiple stale indices in parallel using SNAPSHOT thread pool's workers.
I have added more tests to make sure outage scenario and failover scenario is handled.
Issues Resolved
elastic/elasticsearch#64513
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.