Support cloning of searchable snapshot indices #56595
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Today you can convert a searchable snapshot index back into a regular index by restoring the underlying snapshot, but this is somewhat wasteful if the shards are already in cache since it copies the whole index from the repository again. Instead, we can make use of the locally-cached data by using the clone API to copy the contents of the cache into the layout expected by a regular shard. This commit marks the searchable snapshot's private index settings as `NotCopyableOnResize` so that they are removed by resize operations such as cloning. Cloning a regular index typically hard-links the underlying files rather than copying them, but this is tricky to support in the case of a searchable snapshot so this commit takes the simpler approach of always copying the underlying files.
DaveCTurner
added
>enhancement
:Distributed Coordination/Snapshot/Restore
|
Pinging @elastic/es-distributed (:Distributed/Snapshot/Restore) |
elasticmachine
added
the
Team:Distributed (Obsolete)
tlrx
approved these changes
May 12, 2020
|
Thanks @tlrx |
DaveCTurner
added a commit
that referenced
this pull request
May 13, 2020
Today you can convert a searchable snapshot index back into a regular index by restoring the underlying snapshot, but this is somewhat wasteful if the shards are already in cache since it copies the whole index from the repository again. Instead, we can make use of the locally-cached data by using the clone API to copy the contents of the cache into the layout expected by a regular shard. This commit marks the searchable snapshot's private index settings as `NotCopyableOnResize` so that they are removed by resize operations such as cloning. Cloning a regular index typically hard-links the underlying files rather than copying them, but this is tricky to support in the case of a searchable snapshot so this commit takes the simpler approach of always copying the underlying files.
This was referenced Jul 12, 2021
tlrx
added a commit
that referenced
this pull request
Jul 13, 2021
Today if we try to shrink or to split a searchable snapshot index using the Resize API a new index will be created but can't be assigned, and even if it was assigned it won't work as the number of shards can't be changed and must always match the number of shards from the snapshot. This commit adds some verification to prevent a snapshot backed indices to be resized and if an attempt is made, throw a better error message. Note that cloning is supported since #56595 and in this change we make sure that it is only used to convert the searchable snapshot index back to a regular index. Relates #74977 (comment)
tlrx
added a commit
to tlrx/elasticsearch
that referenced
this pull request
Jul 13, 2021
Today if we try to shrink or to split a searchable snapshot index using the Resize API a new index will be created but can't be assigned, and even if it was assigned it won't work as the number of shards can't be changed and must always match the number of shards from the snapshot. This commit adds some verification to prevent a snapshot backed indices to be resized and if an attempt is made, throw a better error message. Note that cloning is supported since elastic#56595 and in this change we make sure that it is only used to convert the searchable snapshot index back to a regular index. Relates elastic#74977 (comment)
elasticsearchmachine
pushed a commit
that referenced
this pull request
Jul 13, 2021
) Today if we try to shrink or to split a searchable snapshot index using the Resize API a new index will be created but can't be assigned, and even if it was assigned it won't work as the number of shards can't be changed and must always match the number of shards from the snapshot. This commit adds some verification to prevent a snapshot backed indices to be resized and if an attempt is made, throw a better error message. Note that cloning is supported since #56595 and in this change we make sure that it is only used to convert the searchable snapshot index back to a regular index. Relates #74977 (comment)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Today you can convert a searchable snapshot index back into a regular index by
restoring the underlying snapshot, but this is somewhat wasteful if the shards
are already in cache since it copies the whole index from the repository again.
Instead, we can make use of the locally-cached data by using the clone API to
copy the contents of the cache into the layout expected by a regular shard.
This commit marks the searchable snapshot's private index settings as
NotCopyableOnResizeso that they are removed by resize operations such ascloning.
Cloning a regular index typically hard-links the underlying files rather than
copying them, but this is tricky to support in the case of a searchable
snapshot so this commit takes the simpler approach of always copying the
underlying files.