-
Notifications
You must be signed in to change notification settings - Fork 507
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Adds info about new codecs #4571
Merged
Merged
Changes from 3 commits
Commits
Show all changes
13 commits
Select commit
Hold shift + click to select a range
d3548e3
addedinfoaboutcodecs
hdhalter b9d6243
added collapsible toc
hdhalter 14c8d12
Update _tuning-your-cluster/availability-and-recovery/snapshots/snaps…
hdhalter fb29636
Update _tuning-your-cluster/availability-and-recovery/snapshots/snaps…
hdhalter 68aa8dd
Update _tuning-your-cluster/availability-and-recovery/snapshots/snaps…
hdhalter de9a5d3
Update _tuning-your-cluster/availability-and-recovery/snapshots/snaps…
hdhalter 813d978
Update _tuning-your-cluster/availability-and-recovery/snapshots/snaps…
hdhalter 4a09892
Update _tuning-your-cluster/availability-and-recovery/snapshots/snaps…
hdhalter e3b8c88
Update _tuning-your-cluster/availability-and-recovery/snapshots/snaps…
hdhalter 2180da3
Update _tuning-your-cluster/availability-and-recovery/snapshots/snaps…
hdhalter 56523bf
Update _tuning-your-cluster/availability-and-recovery/snapshots/snaps…
hdhalter 7962ff5
Update _tuning-your-cluster/availability-and-recovery/snapshots/snaps…
hdhalter 1ad957e
Update snapshot-restore.md
hdhalter File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -24,14 +24,17 @@ If you need to delete a snapshot, be sure to use the OpenSearch API rather than | |
|
||
--- | ||
|
||
#### Table of contents | ||
1. TOC | ||
<details closed markdown="block"> | ||
<summary> | ||
Table of contents | ||
</summary> | ||
{: .text-delta } | ||
- TOC | ||
{:toc} | ||
|
||
</details> | ||
|
||
--- | ||
|
||
|
||
## Register repository | ||
|
||
Before you can take a snapshot, you have to "register" a snapshot repository. A snapshot repository is just a storage location: a shared file system, Amazon S3, Hadoop Distributed File System (HDFS), Azure Storage, etc. | ||
|
@@ -209,14 +212,14 @@ You specify two pieces of information when you create a snapshot: | |
- Name of your snapshot repository | ||
- Name for the snapshot | ||
|
||
The following snapshot includes all indices and the cluster state: | ||
The following snapshot includes all indexes and the cluster state: | ||
|
||
```json | ||
PUT /_snapshot/my-repository/1 | ||
``` | ||
{% include copy-curl.html %} | ||
|
||
You can also add a request body to include or exclude certain indices or specify other settings: | ||
You can also add a request body to include or exclude certain indexes or specify other settings: | ||
|
||
```json | ||
PUT /_snapshot/my-repository/2 | ||
|
@@ -231,7 +234,7 @@ PUT /_snapshot/my-repository/2 | |
|
||
Request fields | Description | ||
:--- | :--- | ||
`indices` | The indices you want to include in the snapshot. You can use `,` to create a list of indices, `*` to specify an index pattern, and `-` to exclude certain indices. Don't put spaces between items. Default is all indices. | ||
`indices` | The indexes you want to include in the snapshot. You can use `,` to create a list of indexes, `*` to specify an index pattern, and `-` to exclude certain indexes. Don't put spaces between items. Default is all indexes. | ||
`ignore_unavailable` | If an index from the `indices` list doesn't exist, whether to ignore it rather than fail the snapshot. Default is false. | ||
hdhalter marked this conversation as resolved.
Show resolved
Hide resolved
|
||
`include_global_state` | Whether to include cluster state in the snapshot. Default is true. | ||
hdhalter marked this conversation as resolved.
Show resolved
Hide resolved
|
||
`partial` | Whether to allow partial snapshots. Default is false, which fails the entire snapshot if one or more shards fails to store. | ||
hdhalter marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
@@ -306,7 +309,7 @@ POST /_snapshot/my-repository/2/_restore | |
``` | ||
{% include copy-curl.html %} | ||
|
||
Just like when taking a snapshot, you can add a request body to include or exclude certain indices or specify some other settings: | ||
Just like when taking a snapshot, you can add a request body to include or exclude certain indexes or specify some other settings: | ||
|
||
```json | ||
POST /_snapshot/my-repository/2/_restore | ||
|
@@ -330,30 +333,36 @@ POST /_snapshot/my-repository/2/_restore | |
|
||
Request parameters | Description | ||
:--- | :--- | ||
`indices` | The indices you want to restore. You can use `,` to create a list of indices, `*` to specify an index pattern, and `-` to exclude certain indices. Don't put spaces between items. Default is all indices. | ||
`indices` | The indexes you want to restore. You can use `,` to create a list of indexes, `*` to specify an index pattern, and `-` to exclude certain indexes. Don't put spaces between items. Default is all indexes. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Entire list: Should the defaults be in tic marks? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, looks like we add tic marks to the default values. |
||
`ignore_unavailable` | If an index from the `indices` list doesn't exist, whether to ignore it rather than fail the restore operation. Default is false. | ||
`include_global_state` | Whether to restore the cluster state. Default is false. | ||
`include_aliases` | Whether to restore aliases alongside their associated indices. Default is true. | ||
`include_aliases` | Whether to restore aliases alongside their associated indexes. Default is true. | ||
`partial` | Whether to allow the restoration of partial snapshots. Default is false. | ||
`rename_pattern` | If you want to rename indices as you restore them, use this option to specify a regular expression that matches all indices you want to restore. Use capture groups (`()`) to reuse portions of the index name. | ||
`rename_replacement` | If you want to rename indices as you restore them, use this option to specify the replacement pattern. Use `$0` to include the entire matching index name, `$1` to include the content of the first capture group, etc. | ||
`rename_pattern` | If you want to rename indexes as you restore them, use this option to specify a regular expression that matches all indexes you want to restore. Use capture groups (`()`) to reuse portions of the index name. | ||
`rename_replacement` | If you want to rename indexes as you restore them, use this option to specify the replacement pattern. Use `$0` to include the entire matching index name, `$1` to include the content of the first capture group, etc. | ||
hdhalter marked this conversation as resolved.
Show resolved
Hide resolved
|
||
`index_settings` | If you want to change [index settings]({{site.url}}{{site.baseurl}}/api-reference/index-apis/create-index/#index-settings) applied during restore, specify them here. You cannot change `index.number_of_shards`. | ||
`ignore_index_settings` | Rather than explicitly specifying new settings with `index_settings`, you can ignore certain index settings in the snapshot and use the cluster defaults applied during restore. You cannot ignore `index.number_of_shards`, `index.number_of_replicas`, or `index.auto_expand_replicas`. | ||
`storage_type` | `local` indicates that all snapshot metadata and index data will be downloaded to local storage. <br /><br > `remote_snapshot` indicates that snapshot metadata will be downloaded to the cluster, but the remote repository will remain the authoritative store of the index data. Data will be downloaded and cached as necessary to service queries. At least one node in the cluster must be configured with the [search role]({{site.url}}{{site.baseurl}}/security/access-control/users-roles/) in order to restore a snapshot using the type `remote_snapshot`. <br /><br > Defaults to `local`. | ||
|
||
### Conflicts and compatibility | ||
|
||
One way to avoid naming conflicts when restoring indices is to use the `rename_pattern` and `rename_replacement` options. Then, if necessary, you can use the `_reindex` API to combine the two. The simpler way is to delete existing indices prior to restoring from a snapshot. | ||
One way to avoid naming conflicts when restoring indexes is to use the `rename_pattern` and `rename_replacement` options. You can then, if necessary, use the `_reindex` API to combine the two. The simpler way is to delete existing indexes prior to restoring from a snapshot. | ||
|
||
You can use the `_close` API to close existing indices prior to restoring from a snapshot, but the index in the snapshot has to have the same number of shards as the existing index. | ||
You can use the `_close` API to close existing indexes prior to restoring from a snapshot, but the index in the snapshot has to have the same number of shards as the existing index. | ||
|
||
We recommend ceasing write requests to a cluster before restoring from a snapshot, which helps avoid scenarios such as: | ||
|
||
1. You delete an index, which also deletes its alias. | ||
1. A write request to the now-deleted alias creates a new index with the same name as the alias. | ||
1. The alias from the snapshot fails to restore due to a naming conflict with the new index. | ||
|
||
Snapshots are only forward-compatible by one major version. If you have an old snapshot, you can sometimes restore it into an intermediate cluster, reindex all indices, take a new snapshot, and repeat until you arrive at your desired version, but you might find it easier to just manually index your data on the new cluster. | ||
Snapshots are only forward-compatible by one major version. If you have an old snapshot, you can sometimes restore it into an intermediate cluster, reindex all indexes, take a new snapshot, and repeat until you arrive at your desired version, but you might find it easier to just manually index your data on the new cluster. | ||
hdhalter marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
When creating a snapshot, the index codec setting influences both the size of the snapshot and the time required for its creation. If the codec of an index is updated, new snapshots created will utilize the latest codec setting. The resulting snapshot size will reflect the compression characteristics of the latest codec settings. Existing segments included in the snapshot retain their original compression characteristics. | ||
hdhalter marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
If you want to restore the indexes from a snapshot of a cluster to another cluster, it is important to verify that the target cluster supports the codecs used for the segments present in the source snapshot. For example, if the source snapshot uses the `zstd_no_dict` codec with its segments (introduced in OpenSearch 2.9), the user cannot restore to the target cluster unless the target cluster supports this codec. | ||
hdhalter marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
For more information about index codec settings, see [Index codec settings]({{site.url}}{{site.baseurl}}/api-reference/index-apis/create-index/#Index-codec-settings/). | ||
|
||
## Security considerations | ||
|
||
|
@@ -362,7 +371,7 @@ If you're using the Security plugin, snapshots have some additional restrictions | |
- To perform snapshot and restore operations, users must have the built-in `manage_snapshots` role. | ||
- You can't restore snapshots that contain global state or the `.opendistro_security` index. | ||
hdhalter marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
If a snapshot contains global state, you must exclude it when performing the restore. If your snapshot also contains the `.opendistro_security` index, either exclude it or list all the other indices you want to include: | ||
If a snapshot contains global state, you must exclude it when performing the restore. If your snapshot also contains the `.opendistro_security` index, either exclude it or list all the other indexes you want to include: | ||
hdhalter marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
```json | ||
POST /_snapshot/my-repository/3/_restore | ||
|
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I always thought its Indices but interesting to know both plurals are correct