Lazily load soft-deletes for searchable snapshot shards #69203
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dnhatn
reviewed
Feb 18, 2021
server/src/main/java/org/apache/lucene/index/LazySoftDeletesDirectoryReaderWrapper.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/apache/lucene/index/LazySoftDeletesDirectoryReaderWrapper.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/apache/lucene/index/LazySoftDeletesDirectoryReaderWrapper.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/apache/lucene/index/LazySoftDeletesDirectoryReaderWrapper.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/apache/lucene/index/LazySoftDeletesDirectoryReaderWrapper.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/apache/lucene/index/LazySoftDeletesDirectoryReaderWrapper.java
Outdated
Show resolved
Hide resolved
ywelsch
added
:Distributed Coordination/Snapshot/Restore
elasticmachine
added
the
Team:Distributed (Obsolete)
|
Pinging @elastic/es-distributed (Team:Distributed) |
dnhatn
approved these changes
Feb 21, 2021
server/src/main/java/org/apache/lucene/index/LazySoftDeletesDirectoryReaderWrapper.java
Show resolved
Hide resolved
server/src/main/java/org/apache/lucene/index/LazySoftDeletesDirectoryReaderWrapper.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/elasticsearch/index/engine/ReadOnlyEngine.java
Outdated
Show resolved
Hide resolved
jpountz
reviewed
Feb 21, 2021
server/src/main/java/org/apache/lucene/index/LazySoftDeletesDirectoryReaderWrapper.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/apache/lucene/index/LazySoftDeletesDirectoryReaderWrapper.java
Outdated
Show resolved
Hide resolved
| } | ||
| } | ||
|
|
||
| private static class DelegatingCacheHelper implements CacheHelper { |
There was a problem hiding this comment.
I believe that this bit is the only reason why we need to have this class in the oal.index package. Can we find a way to avoid doing cross-JAR package-protected access? (Is there another option than making the CacheKey ctor public?)
There was a problem hiding this comment.
The only other reason except for CacheKey is the call to the PendingSoftDeletes.applySoftDeletes method. That method can easily be reimplemented, however. I don't see a good way to work around the CacheKey constructor not being accessible (safe for introducing even more outrageous hacks), and FWIW we already have some classes like ShuffleForcedMergePolicy and OneMergeHelper in this package.
|
@elasticmachine update branch (test failure unrelated) |
|
@elasticmachine run elasticsearch-ci/1 (yet another unrelated test failure ...) |
jpountz
approved these changes
Feb 22, 2021
ywelsch
added a commit
that referenced
this pull request
Feb 23, 2021
Opening a Lucene index that supports soft-deletes currently creates the liveDocs bitset eagerly. This requires scanning the doc values to materialize the liveDocs bitset from the soft-delete doc values. In order for searchable snapshot shards to be available for searches as quickly as possible (i.e. on recovery, or in case of FrozenEngine whenever a search comes in), they should read as little as possible from the Lucene files. This commit introduces a LazySoftDeletesDirectoryReaderWrapper, a variant of Lucene's SoftDeletesDirectoryReaderWrapper that loads the livedocs bitset lazily on first access. It is special-tailored to ReadOnlyEngine / FrozenEngine as it only operates on non-NRT readers.
ywelsch
added a commit
that referenced
this pull request
Feb 23, 2021
Opening a Lucene index that supports soft-deletes currently creates the liveDocs bitset eagerly. This requires scanning the doc values to materialize the liveDocs bitset from the soft-delete doc values. In order for searchable snapshot shards to be available for searches as quickly as possible (i.e. on recovery, or in case of FrozenEngine whenever a search comes in), they should read as little as possible from the Lucene files. This commit introduces a LazySoftDeletesDirectoryReaderWrapper, a variant of Lucene's SoftDeletesDirectoryReaderWrapper that loads the livedocs bitset lazily on first access. It is special-tailored to ReadOnlyEngine / FrozenEngine as it only operates on non-NRT readers.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Opening a Lucene index that supports soft-deletes currently creates the liveDocs bitset eagerly. This requires scanning the doc values to materialize the liveDocs bitset from the soft-delete doc values. In order for searchable snapshot shards to be available for searches as quickly as possible (i.e. on recovery, or in case of
FrozenEnginewhenever a search comes in), they should read as little as possible from the Lucene files.This PR introduces a
LazySoftDeletesDirectoryReaderWrapper, a variant of Lucene'sSoftDeletesDirectoryReaderWrapperthat loads the livedocs bitset lazily on first access. It is special-tailored toReadOnlyEngine/FrozenEngineas it only operates on non-NRT readers.