Lucene merges should run on the target shard during recovery #10463
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
I moved the mergeScheduleFuture creation to ctor, so now we create it once when the IndexShard is created, not in newEngine. And I fixed EngineMerge to use engineUnsafe and skip merging if engine is currently null... |
|
LGTM |
mikemccand
added a commit
that referenced
this pull request
Apr 7, 2015
This does not affect 2.0, where we let Lucene launch merges normally (#8643). In 1.x, every 1 sec (default), we ask Lucene to kick off any new merges, but we unfortunately don't turn that logic on in the target shard until after recovery has finished. This means if you have a large translog, and/or a smallish index buffer, way too many segments can accumulate in the target shard during recovery, making version lookups slower and slower (OI(N^2)) and possibly causing slow recovery issues like #9226. This fix changes IndexShard to launch merges as soon as the shard is created, so merging runs during recovery. Closes #10463
mikemccand
added a commit
that referenced
this pull request
Apr 7, 2015
This does not affect 2.0, where we let Lucene launch merges normally (#8643). In 1.x, every 1 sec (default), we ask Lucene to kick off any new merges, but we unfortunately don't turn that logic on in the target shard until after recovery has finished. This means if you have a large translog, and/or a smallish index buffer, way too many segments can accumulate in the target shard during recovery, making version lookups slower and slower (OI(N^2)) and possibly causing slow recovery issues like #9226. This fix changes IndexShard to launch merges as soon as the shard is created, so merging runs during recovery. Closes #10463
mikemccand
added a commit
to mikemccand/elasticsearch
that referenced
this pull request
Apr 11, 2015
This does not affect 2.0, where we let Lucene launch merges normally (elastic#8643). In 1.x, every 1 sec (default), we ask Lucene to kick off any new merges, but we unfortunately don't turn that logic on in the target shard until after recovery has finished. This means if you have a large translog, and/or a smallish index buffer, way too many segments can accumulate in the target shard during recovery, making version lookups slower and slower (OI(N^2)) and possibly causing slow recovery issues like elastic#9226. This fix changes IndexShard to launch merges as soon as the shard is created, so merging runs during recovery. Closes elastic#10463
mikemccand
added a commit
that referenced
this pull request
Apr 11, 2015
This does not affect 2.0, where we let Lucene launch merges normally (#8643). In 1.x, every 1 sec (default), we ask Lucene to kick off any new merges, but we unfortunately don't turn that logic on in the target shard until after recovery has finished. This means if you have a large translog, and/or a smallish index buffer, way too many segments can accumulate in the target shard during recovery, making version lookups slower and slower (OI(N^2)) and possibly causing slow recovery issues like #9226. This fix changes IndexShard to launch merges as soon as the shard is created, so merging runs during recovery. Closes #10463
clintongormley
changed the title
mute
pushed a commit
to mute/elasticsearch
that referenced
this pull request
Jul 29, 2015
This does not affect 2.0, where we let Lucene launch merges normally (elastic#8643). In 1.x, every 1 sec (default), we ask Lucene to kick off any new merges, but we unfortunately don't turn that logic on in the target shard until after recovery has finished. This means if you have a large translog, and/or a smallish index buffer, way too many segments can accumulate in the target shard during recovery, making version lookups slower and slower (OI(N^2)) and possibly causing slow recovery issues like elastic#9226. This fix changes IndexShard to launch merges as soon as the shard is created, so merging runs during recovery. Closes elastic#10463
mute
pushed a commit
to mute/elasticsearch
that referenced
this pull request
Jul 29, 2015
This does not affect 2.0, where we let Lucene launch merges normally (elastic#8643). In 1.x, every 1 sec (default), we ask Lucene to kick off any new merges, but we unfortunately don't turn that logic on in the target shard until after recovery has finished. This means if you have a large translog, and/or a smallish index buffer, way too many segments can accumulate in the target shard during recovery, making version lookups slower and slower (OI(N^2)) and possibly causing slow recovery issues like elastic#9226. This fix changes IndexShard to launch merges as soon as the shard is created, so merging runs during recovery. Closes elastic#10463
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is already fixed on 2.0, since we let Lucene launch its own merges again.
But in 1.x, Lucene merges might not run on the target during recovery, causing segment explosion when there are many docs to replay and/or the index buffer is low. This then makes recovery time O(N^2) and can cause issues like #9226.
I just moved launching of the mergeScheduleFuture out of startScheduledTasksIfNeeded (only called once recovery is done) and into createNewEngine. This way whenever the engine is created we also start checking for merges.
I also renamed startScheduledTasksIfNeeded -> startEngineRefresher, and cleaned up a couple unrelated things.