From e88ada887393a463975973f0eb8a43493d8f5a4a Mon Sep 17 00:00:00 2001 From: James Rodewig <40268737+jrodewig@users.noreply.github.com> Date: Tue, 24 Aug 2021 21:36:49 -0400 Subject: [PATCH 01/40] [DOCS] Convert 'Restore a snapshot' to tutorial Updates the `Restore a snapshot` guide to be more tutorial-focused. The current page duplicates reference content from the restore snapshot API docs. I plan to add a tutorial for restoring a full cluster in a later PR. --- docs/build.gradle | 23 ++ docs/reference/redirects.asciidoc | 2 +- .../apis/restore-snapshot-api.asciidoc | 87 ++--- .../reference/snapshot-restore/index.asciidoc | 2 +- .../monitor-snapshot-restore.asciidoc | 27 +- .../restore-snapshot.asciidoc | 355 +++++++++++------- 6 files changed, 295 insertions(+), 201 deletions(-) diff --git a/docs/build.gradle b/docs/build.gradle index ec7168061815e..fd0d453c3feb6 100644 --- a/docs/build.gradle +++ b/docs/build.gradle @@ -1515,6 +1515,29 @@ setups['setup-repository'] = ''' location: buildDir/cluster/shared/repo ''' +// Used by snapshot restore docs +setups['setup-snapshots'] = setups['setup-repository'] + ''' + - do: + bulk: + index: my-index + refresh: true + body: | + {"index":{"_id": "0"}} + {"message": "trying out Elasticsearch", "context": "foo"} + - do: + bulk: + index: logs-my_app-default + refresh: true + body: | + {"create":{"_id": "0"}} + {"message": "trying out Elasticsearch", "context": "foo"} + - do: + snapshot.create: + repository: my_repository + snapshot: my_snapshot + wait_for_completion: true +''' + // Fake sec logs data used by EQL search setups['atomic_red_regsvr32'] = setups['my_data_stream'] + ''' - do: diff --git a/docs/reference/redirects.asciidoc b/docs/reference/redirects.asciidoc index 5e253d39e1ef1..a87c1547b6227 100644 --- a/docs/reference/redirects.asciidoc +++ b/docs/reference/redirects.asciidoc @@ -755,7 +755,7 @@ See <>. [role="exclude",id="_changing_index_settings_during_restore"] ==== Change index settings during restore -See <>. +See <>. [role="exclude",id="restore-snapshot"] === Restore snapshot diff --git a/docs/reference/snapshot-restore/apis/restore-snapshot-api.asciidoc b/docs/reference/snapshot-restore/apis/restore-snapshot-api.asciidoc index ee4e2bb380c97..8370895393ec4 100644 --- a/docs/reference/snapshot-restore/apis/restore-snapshot-api.asciidoc +++ b/docs/reference/snapshot-restore/apis/restore-snapshot-api.asciidoc @@ -64,29 +64,22 @@ POST /_snapshot/my_repository/my_snapshot/_restore [[restore-snapshot-api-prereqs]] ==== {api-prereq-title} -* If the {es} {security-features} are enabled, you must have the `manage` -<> to use this API. +// tag::restore-prereqs[] +* If you use the {es} security features, you must have the `manage` or +`cluster:admin/snapshot/*` cluster privilege to restore a snapshot. -[[restore-snapshot-api-desc]] -==== {api-description-title} +* You can only restore a snapshot to a running cluster with an elected +<>. The snapshot's repository must be +<> and available to the cluster. -Use the restore snapshot API to restore a snapshot of a cluster, including all data streams and indices in the snapshot. If you do not want to restore the entire snapshot, you can select specific data streams or indices to restore. +* The snapshot and cluster versions must be compatible. See +<>. -You can run the restore operation on a cluster that contains an elected -<> and has data nodes with enough capacity to accommodate the snapshot -you are restoring. Existing indices can only be restored if they are -<> and have the same number of shards as the indices in -the snapshot. The restore operation automatically opens restored indices if -they were closed and creates new indices if they do not exist in the cluster. - -If a data stream is restored, its aliases and backing indices are also restored. -Alternatively, you can restore individual backing indices without restoring an -entire data stream. If you restore individual backing indices, they are not -automatically added to any existing data stream. For example, if only the -`.ds-logs-2099.03.08-00003` backing index is restored from a snapshot, it is not -automatically added to the existing `logs` data stream. - -include::{es-ref-dir}/snapshot-restore/restore-snapshot.asciidoc[tag=index-settings-data-stream-warning] +* If you restore a data stream, ensure the cluster contains a +<> with data stream enabled. +Without a matching index template, a data stream cannot roll over or create +backing indices. +// end::restore-prereqs[] [[restore-snapshot-api-path-params]] ==== {api-path-parms-title} @@ -107,7 +100,7 @@ include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=master-timeout] `wait_for_completion`:: (Optional, Boolean) If `true`, the request returns a response when the restore operation completes. The operation is complete when it finishes all attempts to -<<_monitoring_restore_operations,recover primary shards>> for restored indices. +<> for restored indices. This applies even if one or more of the recovery attempts fail. + If `false`, the request returns a response when the restore @@ -118,14 +111,17 @@ operation initializes. Defaults to `false`. ==== {api-request-body-title} `ignore_unavailable`:: -(Optional, Boolean) -If `false`, the request returns an error for any data stream or index that is missing or closed. Defaults to `false`. -+ -If `true`, the request ignores data streams and indices in `indices` that are missing or closed. +(Optional, Boolean) If `true`, the request ignores any index or data stream in +`indices` that's missing from the snapshot. If `false`, the request returns an +error for any missing index or data stream. Defaults to `false`. `ignore_index_settings`:: -(Optional, string) -A comma-separated list of index settings that should not be restored from a snapshot. +(Optional, array of string) Index settings to not restore from the snapshot. You +can't use this option to ignore +<>. ++ +For data streams, this option only applies to restored backing indices. New +backing indices are configured using the data stream's matching index template. `include_aliases`:: (Optional, Boolean) @@ -171,23 +167,25 @@ By default, all available feature states will be restored if `include_global_sta [[restore-snapshot-api-index-settings]] `index_settings`:: -(Optional, string) -A comma-separated list of settings to add or change in all restored indices. Use this parameter to override index settings when restoring snapshots. -+ -For data streams, these index settings are applied to the restored backing -indices. +(Optional, object) Index settings to add or change in restored indices, +including backing indices. You can't use this option to change +<>. + -For more information regarding all the different index-level settings -that you can specify, see -<>. +For data streams, this option only applies to restored backing indices. New +backing indices are configured using the data stream's matching index template. `indices`:: -(Optional, string) -A comma-separated list of data streams and indices to restore from the snapshot. -<> is supported. +(Optional, string) Comma-separated list of indices and data streams to restore. +Supports <>. Defaults to all indices and +data streams in the snapshot, including system indices. ++ +You can only restore an existing index if it's closed. If you restore a data +stream, you also restore it's backing indices. The restore operation +automatically opens restored indices, including backing indices. + -By default, a restore operation includes all data streams and indices in the snapshot. If this -argument is provided, the restore operation only includes the specified data streams and indices. +You can use this option to restore only a specific backing index from a data +stream. However, the restore operation doesn't add the restored backing index to +any existing data stream. [[restore-snapshot-api-partial]] `partial`:: @@ -206,9 +204,14 @@ Defines a rename pattern to apply to restored data streams and indices. Data str The rename pattern is applied as defined by the regular expression that supports referencing the original text, according to the https://docs.oracle.com/javase/8/docs/api/java/util/regex/Matcher.html#appendReplacement-java.lang.StringBuffer-java.lang.String-[`appendReplacement`] logic. + -The request will fail if two or more data streams or indices will be renamed into the same name. +If you rename a data stream, its backing indices are also renamed. For example, +if you rename the `logs-my_app-default` data stream to +`restored-logs-my_app-default`, the backing index +`.ds-logs-my_app-default-2099.03.09-000005` is renamed to +`.ds-restored-logs-my_app-default-2099.03.09-000005`. + -include::{es-ref-dir}/snapshot-restore/restore-snapshot.asciidoc[tag=rename-restored-data-stream-tag] +If this option produces two or more indices or data streams with the same name, +the restore operation fails. [[restore-snapshot-api-rename-replacement]] `rename_replacement`:: diff --git a/docs/reference/snapshot-restore/index.asciidoc b/docs/reference/snapshot-restore/index.asciidoc index 245caed4fb3cc..a40eba86b2ca6 100644 --- a/docs/reference/snapshot-restore/index.asciidoc +++ b/docs/reference/snapshot-restore/index.asciidoc @@ -118,8 +118,8 @@ understand the time requirements before proceeding. include::register-repository.asciidoc[] include::take-snapshot.asciidoc[] -include::restore-snapshot.asciidoc[] include::monitor-snapshot-restore.asciidoc[] include::delete-snapshot.asciidoc[] +include::restore-snapshot.asciidoc[] include::../slm/index.asciidoc[] include::../searchable-snapshots/index.asciidoc[] diff --git a/docs/reference/snapshot-restore/monitor-snapshot-restore.asciidoc b/docs/reference/snapshot-restore/monitor-snapshot-restore.asciidoc index 3e47082b6115c..8364f4a42022b 100644 --- a/docs/reference/snapshot-restore/monitor-snapshot-restore.asciidoc +++ b/docs/reference/snapshot-restore/monitor-snapshot-restore.asciidoc @@ -1,8 +1,5 @@ [[snapshots-monitor-snapshot-restore]] -== Monitor snapshot and restore progress -++++ -Monitor snapshot and restore -++++ +== Monitor snapshot progress Use the <> or the <> to monitor the @@ -132,25 +129,9 @@ if not currently running: GET /_snapshot/my_backup/snapshot_1/_status ----------------------------------- -[discrete] -=== Monitoring restore operations - -The restore process piggybacks on the standard recovery mechanism of -{es}. As a result, standard recovery monitoring services can be used -to monitor the state of restore. When the restore operation starts, the -cluster typically goes into `yellow` state because the restore operation works -by recovering primary shards of the restored indices. After the recovery of the -primary shards is completed, {es} switches to the standard replication -process that creates the required number of replicas. When all required -replicas are created, the cluster switches to the `green` states. - -The cluster health operation provides only a high level status of the restore process. It's possible to get more -detailed insight into the current state of the recovery process by using <> and -<> APIs. - [discrete] [[get-snapshot-stop-snapshot]] -=== Stop snapshot and restore operations +=== Stop snapshot operations To stop a currently running snapshot that was started by mistake or is taking unusually long, use the <>. This operation checks whether the deleted snapshot is currently running. If it is, the delete snapshot operation stops @@ -161,10 +142,6 @@ that snapshot before deleting the snapshot data from the repository. DELETE /_snapshot/my_backup/snapshot_1 ----------------------------------- -The restore operation uses the standard shard recovery mechanism. Therefore, any currently running restore operation can -be canceled by deleting data streams and indices that are being restored. Data for all deleted data streams and indices will be removed -from the cluster as a result of this operation. - [discrete] [[get-snapshot-cluster-blocks]] === Effect of cluster blocks on snapshot and restore diff --git a/docs/reference/snapshot-restore/restore-snapshot.asciidoc b/docs/reference/snapshot-restore/restore-snapshot.asciidoc index 3020cca60e357..bd2a73f21ac80 100644 --- a/docs/reference/snapshot-restore/restore-snapshot.asciidoc +++ b/docs/reference/snapshot-restore/restore-snapshot.asciidoc @@ -1,175 +1,249 @@ [[snapshots-restore-snapshot]] == Restore a snapshot -//// -[source,console] ------------------------------------ -PUT /_snapshot/my_backup -{ - "type": "fs", - "settings": { - "location": "my_backup_location" - } -} +This guide shows you how to restore a snapshot. Snapshots are a convenient way +to transfer data between clusters. You can also restore a snapshot to recover +data after an accidental deletion or a hardware failure. -PUT /_snapshot/my_backup/snapshot_1?wait_for_completion=true ------------------------------------ -// TESTSETUP +In this guide, you'll learn how to: -//// +* Retrieve available snapshots +* Restore an index or data stream from a snapshot +* Monitor the restore operation +* Cancel an ongoing restore -Use the <> to restore -a snapshot of a cluster or specified data streams and indices: +The guide also provides tips for <>. -[source,console] ------------------------------------ -POST /_snapshot/my_backup/snapshot_1/_restore ------------------------------------ -// TEST[s/_restore/_restore?wait_for_completion=true/] +[discrete] +[[restore-snapshot-prereqs]] +=== Prerequisites + +include::apis/restore-snapshot-api.asciidoc[tag=restore-prereqs] -By default, all data streams and indices in the snapshot are restored, but the cluster state is -*not* restored. Use the `indices` parameter to restore only specific data streams or indices. This parameter -supports <>. To include the global cluster state, set -`include_global_state` to `true` in the restore request body. +[discrete] +[[retrieve-snapshot]] +=== Retrieve a snapshot -Because all indices in the snapshot are restored by default, all system indices will be restored -by default as well. +To view a list of available snapshots in {kib}, go to the main menu and click +*Stack Management > Snapshot and Restore*. -[WARNING] -==== -Each data stream requires a matching -<>. The stream uses this -template to create new backing indices. +You can also use the <> to fetch a +list of registered snapshot repositories. -When restoring a data stream, ensure a matching template exists for the stream. -You can do this using one of the following methods: +[source,console] +---- +GET _snapshot +---- +// TEST[setup:setup-snapshots] -* Check for existing templates that match the stream. If no matching template - exists, <>. +Use the <> to retrieve snapshots for a +repository. -* Restore a global cluster state that includes a matching template for the - stream. +[source,console] +---- +GET _snapshot/my_repository/*?filter_path=*.snapshot,*.version_id,*.start_time +---- +// TEST[setup:setup-snapshots] -If no index template matches a data stream, the stream cannot -<> or create new backing indices. -==== +[discrete] +[[restore-index-data-stream]] +=== Restore an index or data stream -The `rename_pattern` -and `rename_replacement` options can be also used to rename data streams and indices on restore -using regular expression that supports referencing the original text, according to the https://docs.oracle.com/javase/8/docs/api/java/util/regex/Matcher.html#appendReplacement-java.lang.StringBuffer-java.lang.String-[`appendReplacement`] logic. +You can restore an index or data stream from a snapshot using {kib}'s *Snapshot +and Restore* feature or the <>. -[[rename-restored-data-stream]] -// tag::rename-restored-data-stream-tag[] -If you rename a restored data stream, its backing indices are also -renamed. For example, if you rename the `logs` data stream to `restored-logs`, -the backing index `.ds-logs-2099.03.09-000005` is renamed to -`.ds-restored-logs-2099.03.09-000005`. +You can only restore an existing index if it's closed. Use one of the following +methods to avoid conflicts with existing indices or data streams: -[WARNING] -==== -If you rename a restored stream, ensure an index template matches the new stream -name. If no index template matches the stream, it cannot -<> or create new backing indices. -==== -// end::rename-restored-data-stream-tag[] +* <> +* <> +* <> -To prevent aliases from being restored with their associated data streams and -indices, set `include_aliases` to `false`. +[discrete] +[[delete-restore]] +==== Delete and restore + +The simplest way to avoid conflicts is to delete an existing index or data +stream before restoring it. To prevent the accidental recreation of the index or +data stream, we recommend you temporarily stop all indexing until the restore +operation is complete. + +// Only display warning in 7.x. +// In 7.x, `action.destructive_requires_name` defaults to false. +ifeval::[ "{major-version}" == "7.x" ] +WARNING: Don't use the <> to target the +`*` or `.*` wildcard expressions. If you use {es}'s security features, this will +delete system indices required for authentication. To delete all regular +indices, use `*,-.*`. +endif::[] [source,console] ------------------------------------ -POST /_snapshot/my_backup/snapshot_1/_restore +---- +# Delete an index +DELETE my-index + +# Delete a data stream +DELETE _data_stream/logs-my_app-default + +# Restore the index and data stream from a snapshot +POST _snapshot/my_repository/my_snapshot/_restore { - "indices": "data_stream_1,index_1,index_2", - "ignore_unavailable": true, - "include_global_state": false, <1> - "rename_pattern": "index_(.+)", - "rename_replacement": "restored_index_$1", - "include_aliases": false + "indices": "my-index,logs-my_app-default" } ------------------------------------ -// TEST[continued] - -<1> By default, `include_global_state` is `false`, meaning the snapshot's -cluster state and feature states are not restored. -+ -If `true` then the restore operation merges the legacy index templates in your -cluster with the templates contained in the snapshot, replacing any existing -ones whose name matches one in the snapshot. It completely removes all -persistent settings, non-legacy index templates, ingest pipelines and -{ilm-init} lifecycle policies that exist in your cluster and replaces them with -the corresponding items from the snapshot. - -The restore operation must be performed on a functioning cluster. However, an -existing index can only be restored if it's <> and -has the same number of shards as the index in the snapshot. The restore -operation automatically opens restored indices if they were closed and creates -new indices if they didn't exist in the cluster. - -If a data stream is restored, its backing indices are also restored. The restore -operation automatically opens restored backing indices if they were closed. - -In addition to entire data streams, you can restore only specific backing -indices from a snapshot. However, restored backing indices are not automatically -added to any existing data streams. For example, if only the -`.ds-logs-2099.03.08-000003` backing index is restored from a snapshot, it is -not automatically added to the existing `logs` data stream. +---- +// TEST[setup:setup-snapshots] +// TEST[s/_restore/_restore?wait_for_completion=true/] [discrete] -=== Partial restore +[[close-restore]] +==== Close and restore + +You can restore an existing index if it's closed. However, the index in the +snapshot must have the same number of primary shards. You can't close a data +stream. + +// Only display warning in 7.x. +// In 7.x, `action.destructive_requires_name` defaults to false. +ifeval::["{major-version}" == "7.x"] +WARNING: Don't use the <> to target the `*` or +`.*` wildcard expressions. If you use {es}'s security features, this will close +system indices required for authentication. To close all regular indices, use +`*,-.*`. +endif::[] -By default, the entire restore operation will fail if one or more indices or backing indices participating in the operation don't have -snapshots of all shards available. It can occur if some shards failed to snapshot for example. It is still possible to -restore such indices by setting `partial` to `true`. Please note, that only successfully snapshotted shards will be -restored in this case and all missing shards will be recreated empty. +[source,console] +---- +# Close an index +POST my-index/_close +# Restore the index from a snapshot +POST _snapshot/my_repository/my_snapshot/_restore +{ + "indices": "my-index" +} +---- +// TEST[setup:setup-snapshots] +// TEST[s/_restore/_restore?wait_for_completion=true/] [discrete] -[[change-index-settings-during-restore]] -=== Changing index settings during restore +[[rename-on-restore]] +==== Rename on restore + +You can rename indices and data streams you restore. Before you start, ensure +the cluster has enough capacity to store both the existing and restored data. -Use the <> parameter -to override index settings during the restore process. For example, the -following request will restore the index `index_1` without creating any -replicas while switching back to the default refresh interval: +The following restore snapshot API request prepends `restored-` to the name of +any restored index or data stream. [source,console] ------------------------------------ -POST /_snapshot/my_backup/snapshot_1/_restore +---- +POST _snapshot/my_repository/my_snapshot/_restore { - "indices": "index_1", - "ignore_unavailable": true, - "index_settings": { - "index.number_of_replicas": 0 + "indices": "my-index,logs-my_app-default", + "rename_pattern": "(.+)", + "rename_replacement": "restored-$1" +} +---- +// TEST[setup:setup-snapshots] +// TEST[s/_restore/_restore?wait_for_completion=true/] + +When the restore operation is complete, you can delete the original indices and +data streams. You can then use a <> to rename the restored +indices and data streams. + +[source,console] +---- +# Delete the original index +DELETE my-index + +# Reindex the restored index to rename it +POST _reindex +{ + "source": { + "index": "restored-my-index" + }, + "dest": { + "index": "my-index" + } +} + +# Delete the original data stream +DELETE _data_stream/logs-my_app-default + +# Reindex the restored data stream to rename it +POST _reindex +{ + "source": { + "index": "restored-logs-my_app-default" }, - "ignore_index_settings": [ - "index.refresh_interval" - ] + "dest": { + "index": "logs-my_app-default", + "op_type": "create" + } } ------------------------------------ +---- // TEST[continued] -NOTE: Some settings such as `index.number_of_shards` cannot be changed during the restore operation. +[discrete] +[[monitor-restore]] +=== Monitor a restore + +The restore operation uses the <> to restore +an index's primary shards from a snapshot. While the restore operation +recovers primary shards, the cluster will have a `yellow` +<>. + +After all primary shards are recovered, the replication process creates and +distributes replicas across eligible data nodes. When replication is complete, +the cluster health status typically becomes `green`. + +You can monitor the cluster health status using the <>. -For data streams, these index settings are applied to the restored backing -indices. +[source,console] +---- +GET _cluster/health +---- + +To get detailed information about ongoing shard recoveries, use the +<>. -// tag::index-settings-data-stream-warning[] -[IMPORTANT] -==== -The `index_settings` and `ignore_index_settings` parameters affect -restored backing indices only. New backing indices created for a stream use the index -settings specified in the stream's matching -<>. +[source,console] +---- +GET my-index/_recovery +---- +// TEST[setup:setup-snapshots] -If you change index settings during a restore, we recommend you make similar -changes in the stream's matching index template. This ensures new backing -indices created for the stream use the same index settings. -==== -// end::index-settings-data-stream-warning[] +To get detailed information about unassigned primary shards or replicas +resulting from a restore operation, use the <>. + +[source,console] +---- +GET _cluster/allocation/explain +---- +// TEST[catch:bad_request] + +[discrete] +[[cancel-restore]] +=== Cancel a restore + +You can delete an index or data stream to cancel its ongoing restore. + +[source,console] +---- +# Delete an index +DELETE my-index + +# Delete a data stream +DELETE _data_stream/logs-my_app-default +---- +// TEST[setup:setup-snapshots] [discrete] -=== Restoring to a different cluster +[[restore-different-cluster]] +=== Restore to a different cluster The information stored in a snapshot is not tied to a particular cluster or a cluster name. Therefore, it's possible to restore a snapshot created from one cluster into another cluster by registering the repository that contains the snapshot in the new cluster and starting the restore process. @@ -179,9 +253,26 @@ The topology of the new cluster does not have to match the cluster where the sna If the new cluster has a smaller capacity, consider the following changes: * Ensure that the new cluster has enough capacity to store all data streams and indices in the snapshot. +* Restore fewer indices and data streams from the snapshot. * Change the index settings during the restore operation to reduce the <>. -* Use the `indices` parameter to choose only specific data streams or indices to restore. ++ +For example, the following restore snapshot API request uses the +`index_settings` option to set `index.number_of_replicas` to `1`. ++ +[source,console] +---- +POST _snapshot/my_repository/my_snapshot/_restore +{ + "indices": "my-index,logs-my_app-default", + "index_settings": { + "index.number_of_replicas": 1 + } +} +---- +// TEST[setup:setup-snapshots] +// TEST[s/^/DELETE my-index\nDELETE _data_stream\/logs-my_app-default\n/] +// TEST[s/_restore/_restore?wait_for_completion=true/] If indices or backing indices in the original cluster were assigned to particular nodes using <>, the same rules will be enforced in the new cluster. If the new cluster does not contain nodes with appropriate attributes that a restored index can be allocated on, the From 6d17aea3bdc96e5a34b2650d7b97a7177dcb1432 Mon Sep 17 00:00:00 2001 From: James Rodewig <40268737+jrodewig@users.noreply.github.com> Date: Wed, 25 Aug 2021 13:37:10 -0400 Subject: [PATCH 02/40] Edit I forgot to commit --- .../snapshot-restore/apis/restore-snapshot-api.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/snapshot-restore/apis/restore-snapshot-api.asciidoc b/docs/reference/snapshot-restore/apis/restore-snapshot-api.asciidoc index 8370895393ec4..5b47863cba259 100644 --- a/docs/reference/snapshot-restore/apis/restore-snapshot-api.asciidoc +++ b/docs/reference/snapshot-restore/apis/restore-snapshot-api.asciidoc @@ -77,7 +77,7 @@ POST /_snapshot/my_repository/my_snapshot/_restore * If you restore a data stream, ensure the cluster contains a <> with data stream enabled. -Without a matching index template, a data stream cannot roll over or create +Without a matching index template, a data stream can't roll over or create backing indices. // end::restore-prereqs[] From 2104ace19fb31e85f30ac0b2b521b49f31edb1b0 Mon Sep 17 00:00:00 2001 From: James Rodewig <40268737+jrodewig@users.noreply.github.com> Date: Wed, 25 Aug 2021 14:27:25 -0400 Subject: [PATCH 03/40] Better instructs for unassigned shards --- .../restore-snapshot.asciidoc | 28 +++++++++++++++---- 1 file changed, 23 insertions(+), 5 deletions(-) diff --git a/docs/reference/snapshot-restore/restore-snapshot.asciidoc b/docs/reference/snapshot-restore/restore-snapshot.asciidoc index bd2a73f21ac80..6379f9b2a73a2 100644 --- a/docs/reference/snapshot-restore/restore-snapshot.asciidoc +++ b/docs/reference/snapshot-restore/restore-snapshot.asciidoc @@ -215,15 +215,33 @@ GET my-index/_recovery ---- // TEST[setup:setup-snapshots] -To get detailed information about unassigned primary shards or replicas -resulting from a restore operation, use the <>. +To view any unassigned shards, use the <>. [source,console] ---- -GET _cluster/allocation/explain +GET _cat/shards?v=true&h=index,shard,prirep,state,node,unassigned.reason&s=state ---- -// TEST[catch:bad_request] + +Unassigned shards have a `state` of `UNASSIGNED`. The `prirep` value is `p` for +primary shards and `r` for replicas. The `unassigned.reason` describes why the +shard remains unassigned. + +To get a more in-depth explanation of an unassigned shard's allocation status, +use the <>. + +[source,console] +---- +GET _cluster/allocation/explain?filter_path=index,node_allocation_decisions.node_name,node_allocation_decisions.deciders.* +{ + "index": "my-index", + "shard": 0, + "primary": false, + "current_node": "my-node" +} +---- +// TEST[s/^/PUT my-index\n/] +// TEST[s/"primary": false,/"primary": false/] +// TEST[s/"current_node": "my-node"//] [discrete] [[cancel-restore]] From b73900a40cecb36b551b26f3923561bde92b7394 Mon Sep 17 00:00:00 2001 From: James Rodewig <40268737+jrodewig@users.noreply.github.com> Date: Wed, 25 Aug 2021 16:59:43 -0400 Subject: [PATCH 04/40] Clarify --- docs/reference/snapshot-restore/restore-snapshot.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/snapshot-restore/restore-snapshot.asciidoc b/docs/reference/snapshot-restore/restore-snapshot.asciidoc index 6379f9b2a73a2..a1fcf3ef385b5 100644 --- a/docs/reference/snapshot-restore/restore-snapshot.asciidoc +++ b/docs/reference/snapshot-restore/restore-snapshot.asciidoc @@ -3,7 +3,7 @@ This guide shows you how to restore a snapshot. Snapshots are a convenient way to transfer data between clusters. You can also restore a snapshot to recover -data after an accidental deletion or a hardware failure. +an index or data stream after deletion or a hardware failure. In this guide, you'll learn how to: From fe95c6b9176aea1289c2cdc6bbef6740e80da015 Mon Sep 17 00:00:00 2001 From: James Rodewig <40268737+jrodewig@users.noreply.github.com> Date: Wed, 25 Aug 2021 17:00:21 -0400 Subject: [PATCH 05/40] Clarify --- docs/reference/snapshot-restore/restore-snapshot.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/snapshot-restore/restore-snapshot.asciidoc b/docs/reference/snapshot-restore/restore-snapshot.asciidoc index a1fcf3ef385b5..66c23a777c7f9 100644 --- a/docs/reference/snapshot-restore/restore-snapshot.asciidoc +++ b/docs/reference/snapshot-restore/restore-snapshot.asciidoc @@ -3,7 +3,7 @@ This guide shows you how to restore a snapshot. Snapshots are a convenient way to transfer data between clusters. You can also restore a snapshot to recover -an index or data stream after deletion or a hardware failure. +indices after deletion or a hardware failure. In this guide, you'll learn how to: From 9852a93ab2b032cb8982fa280dc9e3cea292c61b Mon Sep 17 00:00:00 2001 From: James Rodewig <40268737+jrodewig@users.noreply.github.com> Date: Thu, 26 Aug 2021 11:01:37 -0400 Subject: [PATCH 06/40] Note about existing data streams --- .../snapshot-restore/apis/restore-snapshot-api.asciidoc | 7 ++++--- docs/reference/snapshot-restore/restore-snapshot.asciidoc | 5 +++-- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/docs/reference/snapshot-restore/apis/restore-snapshot-api.asciidoc b/docs/reference/snapshot-restore/apis/restore-snapshot-api.asciidoc index 5b47863cba259..8ef76acc86810 100644 --- a/docs/reference/snapshot-restore/apis/restore-snapshot-api.asciidoc +++ b/docs/reference/snapshot-restore/apis/restore-snapshot-api.asciidoc @@ -179,9 +179,10 @@ backing indices are configured using the data stream's matching index template. Supports <>. Defaults to all indices and data streams in the snapshot, including system indices. + -You can only restore an existing index if it's closed. If you restore a data -stream, you also restore it's backing indices. The restore operation -automatically opens restored indices, including backing indices. +You can only restore an existing index if it's closed. You can't restore an +existing data stream. If you restore a data stream, you also restore it's +backing indices. The restore operation automatically opens restored indices, +including backing indices. + You can use this option to restore only a specific backing index from a data stream. However, the restore operation doesn't add the restored backing index to diff --git a/docs/reference/snapshot-restore/restore-snapshot.asciidoc b/docs/reference/snapshot-restore/restore-snapshot.asciidoc index 66c23a777c7f9..0415e9d80a446 100644 --- a/docs/reference/snapshot-restore/restore-snapshot.asciidoc +++ b/docs/reference/snapshot-restore/restore-snapshot.asciidoc @@ -53,8 +53,9 @@ GET _snapshot/my_repository/*?filter_path=*.snapshot,*.version_id,*.start_time You can restore an index or data stream from a snapshot using {kib}'s *Snapshot and Restore* feature or the <>. -You can only restore an existing index if it's closed. Use one of the following -methods to avoid conflicts with existing indices or data streams: +You can only restore an existing index if it's closed. You can't restore an +existing data stream. Use one of the following methods to avoid conflicts with +existing indices or data streams: * <> * <> From 9fdf13e538a0b5cbe8993812d9ee1a289e37f597 Mon Sep 17 00:00:00 2001 From: James Rodewig <40268737+jrodewig@users.noreply.github.com> Date: Thu, 26 Aug 2021 11:22:02 -0400 Subject: [PATCH 07/40] Fix typo --- .../snapshot-restore/apis/restore-snapshot-api.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/snapshot-restore/apis/restore-snapshot-api.asciidoc b/docs/reference/snapshot-restore/apis/restore-snapshot-api.asciidoc index 8ef76acc86810..98c39b8e3bd0c 100644 --- a/docs/reference/snapshot-restore/apis/restore-snapshot-api.asciidoc +++ b/docs/reference/snapshot-restore/apis/restore-snapshot-api.asciidoc @@ -180,7 +180,7 @@ Supports <>. Defaults to all indices and data streams in the snapshot, including system indices. + You can only restore an existing index if it's closed. You can't restore an -existing data stream. If you restore a data stream, you also restore it's +existing data stream. If you restore a data stream, you also restore its backing indices. The restore operation automatically opens restored indices, including backing indices. + From 447bd920bac9300e25a595cdb47442e2c06a9d62 Mon Sep 17 00:00:00 2001 From: James Rodewig <40268737+jrodewig@users.noreply.github.com> Date: Thu, 26 Aug 2021 17:11:55 -0400 Subject: [PATCH 08/40] Better get snapshot query params --- docs/reference/snapshot-restore/restore-snapshot.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/snapshot-restore/restore-snapshot.asciidoc b/docs/reference/snapshot-restore/restore-snapshot.asciidoc index 0415e9d80a446..7a914aeea3aa5 100644 --- a/docs/reference/snapshot-restore/restore-snapshot.asciidoc +++ b/docs/reference/snapshot-restore/restore-snapshot.asciidoc @@ -42,7 +42,7 @@ repository. [source,console] ---- -GET _snapshot/my_repository/*?filter_path=*.snapshot,*.version_id,*.start_time +GET _snapshot/my_repository/*?verbose=false ---- // TEST[setup:setup-snapshots] From e9c441f3e15906b01de1bda40bf8081cdfabb30b Mon Sep 17 00:00:00 2001 From: James Rodewig <40268737+jrodewig@users.noreply.github.com> Date: Thu, 26 Aug 2021 17:22:54 -0400 Subject: [PATCH 09/40] more realistic snapshot name --- docs/build.gradle | 2 +- docs/reference/snapshot-restore/restore-snapshot.asciidoc | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/build.gradle b/docs/build.gradle index fd0d453c3feb6..f84bc91cf7489 100644 --- a/docs/build.gradle +++ b/docs/build.gradle @@ -1534,7 +1534,7 @@ setups['setup-snapshots'] = setups['setup-repository'] + ''' - do: snapshot.create: repository: my_repository - snapshot: my_snapshot + snapshot: my_snapshot_2099.05.06 wait_for_completion: true ''' diff --git a/docs/reference/snapshot-restore/restore-snapshot.asciidoc b/docs/reference/snapshot-restore/restore-snapshot.asciidoc index 7a914aeea3aa5..1f17c02266fff 100644 --- a/docs/reference/snapshot-restore/restore-snapshot.asciidoc +++ b/docs/reference/snapshot-restore/restore-snapshot.asciidoc @@ -88,7 +88,7 @@ DELETE my-index DELETE _data_stream/logs-my_app-default # Restore the index and data stream from a snapshot -POST _snapshot/my_repository/my_snapshot/_restore +POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore { "indices": "my-index,logs-my_app-default" } @@ -119,7 +119,7 @@ endif::[] POST my-index/_close # Restore the index from a snapshot -POST _snapshot/my_repository/my_snapshot/_restore +POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore { "indices": "my-index" } @@ -139,7 +139,7 @@ any restored index or data stream. [source,console] ---- -POST _snapshot/my_repository/my_snapshot/_restore +POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore { "indices": "my-index,logs-my_app-default", "rename_pattern": "(.+)", @@ -281,7 +281,7 @@ For example, the following restore snapshot API request uses the + [source,console] ---- -POST _snapshot/my_repository/my_snapshot/_restore +POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore { "indices": "my-index,logs-my_app-default", "index_settings": { From 986850c69e5c9bf7a2534ffccd25906d38401276 Mon Sep 17 00:00:00 2001 From: James Rodewig <40268737+jrodewig@users.noreply.github.com> Date: Tue, 7 Sep 2021 14:13:32 -0400 Subject: [PATCH 10/40] Address review feedback --- .../restore-snapshot.asciidoc | 100 ++++++++++++++---- 1 file changed, 79 insertions(+), 21 deletions(-) diff --git a/docs/reference/snapshot-restore/restore-snapshot.asciidoc b/docs/reference/snapshot-restore/restore-snapshot.asciidoc index 1f17c02266fff..654af5376900e 100644 --- a/docs/reference/snapshot-restore/restore-snapshot.asciidoc +++ b/docs/reference/snapshot-restore/restore-snapshot.asciidoc @@ -53,9 +53,9 @@ GET _snapshot/my_repository/*?verbose=false You can restore an index or data stream from a snapshot using {kib}'s *Snapshot and Restore* feature or the <>. -You can only restore an existing index if it's closed. You can't restore an -existing data stream. Use one of the following methods to avoid conflicts with -existing indices or data streams: +You can't restore an existing open index or an existing data stream. To avoid +conflicts with existing indices and data streams, use one of the following +methods: * <> * <> @@ -74,7 +74,7 @@ operation is complete. // In 7.x, `action.destructive_requires_name` defaults to false. ifeval::[ "{major-version}" == "7.x" ] WARNING: Don't use the <> to target the -`*` or `.*` wildcard expressions. If you use {es}'s security features, this will +`*` or `.*` wildcard expression. If you use {es}'s security features, this will delete system indices required for authentication. To delete all regular indices, use `*,-.*`. endif::[] @@ -100,39 +100,67 @@ POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore [[close-restore]] ==== Close and restore -You can restore an existing index if it's closed. However, the index in the -snapshot must have the same number of primary shards. You can't close a data -stream. +You can only restore an existing index if it's closed. Typically, you'd only use +this method if you've already closed an index for other reasons. Otherwise, the +results are the same as the <> method. + +You can close an index using the <>. +You can't close a data stream. // Only display warning in 7.x. // In 7.x, `action.destructive_requires_name` defaults to false. ifeval::["{major-version}" == "7.x"] -WARNING: Don't use the <> to target the `*` or -`.*` wildcard expressions. If you use {es}'s security features, this will close -system indices required for authentication. To close all regular indices, use -`*,-.*`. +WARNING: Don't use the close index API to target the `*` or `.*` wildcard +expression. If you use {es}'s security features, this will close system indices +required for authentication. To close all regular indices, use `*,-.*`. endif::[] [source,console] ---- -# Close an index POST my-index/_close +---- +// TEST[setup:setup-snapshots] + +To restore this closed index, the index in the snapshot must have the same +number of primary shards. You can check the primary shard counts using the +<> and <> APIs. + +[source,console] +---- +# Get the primary shard count for the closed index +GET /_cat/indices/my-index?v=true&h=index,pri,status,health + +# Get the primary shard count for the index in the snapshot +GET _snapshot/my_repository/my_snapshot_2099.05.06?index_details=true&filter_path=*.index_details.my-index.shard_count +---- +// TEST[continued] -# Restore the index from a snapshot +If the primary shard counts match, you can restore the closed index. Restoring +the index deletes its existing data and automatically opens it. + +[source,console] +---- POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore { "indices": "my-index" } ---- -// TEST[setup:setup-snapshots] +// TEST[continued] // TEST[s/_restore/_restore?wait_for_completion=true/] [discrete] [[rename-on-restore]] ==== Rename on restore -You can rename indices and data streams you restore. Before you start, ensure -the cluster has enough capacity to store both the existing and restored data. +You can rename indices and data streams you restore. This lets you restore data +from a snapshot without deleting data. + +You can use this method to compare a cluster's existing data to historical data +from a snapshot. For example, you could review changes made to a document after +an accidental update. + +Before you start, ensure the cluster has enough capacity for both the existing +and restored data. The following restore snapshot API request prepends `restored-` to the name of any restored index or data stream. @@ -149,9 +177,9 @@ POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore // TEST[setup:setup-snapshots] // TEST[s/_restore/_restore?wait_for_completion=true/] -When the restore operation is complete, you can delete the original indices and -data streams. You can then use a <> to rename the restored -indices and data streams. +When the restore operation is complete, you can compare the original and +restored data. If you no longer need an original index or data stream, you can +delete it and use a <> to rename the restored one. [source,console] ---- @@ -248,7 +276,8 @@ GET _cluster/allocation/explain?filter_path=index,node_allocation_decisions.node [[cancel-restore]] === Cancel a restore -You can delete an index or data stream to cancel its ongoing restore. +You can delete an index or data stream to cancel its ongoing restore. This also +deletes any existing data for the index or data stream. [source,console] ---- @@ -264,10 +293,16 @@ DELETE _data_stream/logs-my_app-default [[restore-different-cluster]] === Restore to a different cluster +TIP: {ess} can help you restore snapshots from other deployment. See +{cloud}/ec-restoring-snapshots.html#ec-restore-across-clusters[Restore across +clusters]. + The information stored in a snapshot is not tied to a particular cluster or a cluster name. Therefore, it's possible to restore a snapshot created from one cluster into another cluster by registering the repository that contains the snapshot in the new cluster and starting the restore process. -The topology of the new cluster does not have to match the cluster where the snapshot was created. However, the version of the new cluster must be the same or only one major version newer than the cluster that was used to create the snapshot. For example, you can restore a 5.x snapshot to a 6.x cluster, but not a 5.x snapshot to a 7.x cluster. +The topology of the new cluster does not have to match the cluster where the +snapshot was created. However, the versions must be +<>. If the new cluster has a smaller capacity, consider the following changes: @@ -300,3 +335,26 @@ index will not be successfully restored unless these index allocation settings a The restore operation also checks that restored persistent settings are compatible with the current cluster to avoid accidentally restoring incompatible settings. If you need to restore a snapshot with incompatible persistent settings, try restoring it without the <>. + +[discrete] +[[troubleshoot-snapshot-restore]] +=== Troubleshoot snapshot restore errors + +Here's how to resolve common errors returned by restore requests. + +[discrete] +==== Cannot restore index [] because an open index with same name already exists in the cluster + +The request is trying to restore an open index or data stream that already +exists in the cluster. To avoid this error, use one of the following methods: + +* <> +* <> +* <> + +[discrete] +==== Cannot restore index [] with [x] shards from a snapshot of index [] with [y] shards + +If you restore an existing closed index, the index in the snapshot must have the +same number of primary shards. Try <> or <>. From bdd420126852f3272cfcdf0e1a676f461dc41449 Mon Sep 17 00:00:00 2001 From: James Rodewig <40268737+jrodewig@users.noreply.github.com> Date: Tue, 7 Sep 2021 14:22:00 -0400 Subject: [PATCH 11/40] Edits --- .../snapshot-restore/restore-snapshot.asciidoc | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-) diff --git a/docs/reference/snapshot-restore/restore-snapshot.asciidoc b/docs/reference/snapshot-restore/restore-snapshot.asciidoc index 654af5376900e..393ea831b11db 100644 --- a/docs/reference/snapshot-restore/restore-snapshot.asciidoc +++ b/docs/reference/snapshot-restore/restore-snapshot.asciidoc @@ -121,7 +121,7 @@ POST my-index/_close ---- // TEST[setup:setup-snapshots] -To restore this closed index, the index in the snapshot must have the same +To restore the closed index, the index in the snapshot must have the same number of primary shards. You can check the primary shard counts using the <> and <> APIs. @@ -152,12 +152,9 @@ POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore [[rename-on-restore]] ==== Rename on restore -You can rename indices and data streams you restore. This lets you restore data -from a snapshot without deleting data. - -You can use this method to compare a cluster's existing data to historical data -from a snapshot. For example, you could review changes made to a document after -an accidental update. +You can rename indices and data streams you restore. Use this method if you want +to avoid deleting existing data. This lets you compare existing data to +historical data from a snapshot. Before you start, ensure the cluster has enough capacity for both the existing and restored data. @@ -355,6 +352,6 @@ exists in the cluster. To avoid this error, use one of the following methods: [discrete] ==== Cannot restore index [] with [x] shards from a snapshot of index [] with [y] shards -If you restore an existing closed index, the index in the snapshot must have the +To restore an existing closed index, the index in the snapshot must have the same number of primary shards. Try <> or <>. From cfcdd3b60c9be70ff53f6443ae6e1dfb323c1a0a Mon Sep 17 00:00:00 2001 From: James Rodewig <40268737+jrodewig@users.noreply.github.com> Date: Tue, 7 Sep 2021 14:25:42 -0400 Subject: [PATCH 12/40] reword --- docs/reference/snapshot-restore/restore-snapshot.asciidoc | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/reference/snapshot-restore/restore-snapshot.asciidoc b/docs/reference/snapshot-restore/restore-snapshot.asciidoc index 393ea831b11db..025e01e79a8ff 100644 --- a/docs/reference/snapshot-restore/restore-snapshot.asciidoc +++ b/docs/reference/snapshot-restore/restore-snapshot.asciidoc @@ -152,9 +152,7 @@ POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore [[rename-on-restore]] ==== Rename on restore -You can rename indices and data streams you restore. Use this method if you want -to avoid deleting existing data. This lets you compare existing data to -historical data from a snapshot. +You can rename indices and data streams you restore. Use this method if you want to avoid deleting existing data or if you want to compare existing data to historical data from a snapshot. Before you start, ensure the cluster has enough capacity for both the existing and restored data. From 7411d947189c65a75754c4a433765b9988ff585d Mon Sep 17 00:00:00 2001 From: James Rodewig <40268737+jrodewig@users.noreply.github.com> Date: Tue, 7 Sep 2021 14:45:53 -0400 Subject: [PATCH 13/40] reword --- docs/reference/snapshot-restore/restore-snapshot.asciidoc | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/reference/snapshot-restore/restore-snapshot.asciidoc b/docs/reference/snapshot-restore/restore-snapshot.asciidoc index 025e01e79a8ff..a1b6b99ee1faf 100644 --- a/docs/reference/snapshot-restore/restore-snapshot.asciidoc +++ b/docs/reference/snapshot-restore/restore-snapshot.asciidoc @@ -3,7 +3,7 @@ This guide shows you how to restore a snapshot. Snapshots are a convenient way to transfer data between clusters. You can also restore a snapshot to recover -indices after deletion or a hardware failure. +indices and data streams after deletion or a hardware failure. In this guide, you'll learn how to: @@ -152,7 +152,9 @@ POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore [[rename-on-restore]] ==== Rename on restore -You can rename indices and data streams you restore. Use this method if you want to avoid deleting existing data or if you want to compare existing data to historical data from a snapshot. +You can rename indices and data streams you restore. You typically use this +method to compare existing data to historical data from a snapshot. This method +doesn't require the deletion or overwrite of existing data. Before you start, ensure the cluster has enough capacity for both the existing and restored data. From e0562b04329e878e33f77d8f5dd537f638208921 Mon Sep 17 00:00:00 2001 From: James Rodewig <40268737+jrodewig@users.noreply.github.com> Date: Tue, 7 Sep 2021 14:46:55 -0400 Subject: [PATCH 14/40] reorder --- docs/reference/snapshot-restore/restore-snapshot.asciidoc | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/reference/snapshot-restore/restore-snapshot.asciidoc b/docs/reference/snapshot-restore/restore-snapshot.asciidoc index a1b6b99ee1faf..ae97451dfe4af 100644 --- a/docs/reference/snapshot-restore/restore-snapshot.asciidoc +++ b/docs/reference/snapshot-restore/restore-snapshot.asciidoc @@ -152,9 +152,9 @@ POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore [[rename-on-restore]] ==== Rename on restore -You can rename indices and data streams you restore. You typically use this -method to compare existing data to historical data from a snapshot. This method -doesn't require the deletion or overwrite of existing data. +You can rename indices and data streams you restore. This method doesn't require +the deletion or overwrite of any existing data. You typically use this method to +compare existing data to historical data from a snapshot. Before you start, ensure the cluster has enough capacity for both the existing and restored data. From 4b37a701cc8951610aa11bc7c0719ee16967db0d Mon Sep 17 00:00:00 2001 From: James Rodewig <40268737+jrodewig@users.noreply.github.com> Date: Tue, 7 Sep 2021 15:19:04 -0400 Subject: [PATCH 15/40] reword --- .../reference/snapshot-restore/restore-snapshot.asciidoc | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/reference/snapshot-restore/restore-snapshot.asciidoc b/docs/reference/snapshot-restore/restore-snapshot.asciidoc index ae97451dfe4af..0c7ece5436082 100644 --- a/docs/reference/snapshot-restore/restore-snapshot.asciidoc +++ b/docs/reference/snapshot-restore/restore-snapshot.asciidoc @@ -13,7 +13,8 @@ In this guide, you'll learn how to: * Cancel an ongoing restore The guide also provides tips for <>. +another cluster>> and <>. [discrete] [[restore-snapshot-prereqs]] @@ -152,9 +153,9 @@ POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore [[rename-on-restore]] ==== Rename on restore -You can rename indices and data streams you restore. This method doesn't require -the deletion or overwrite of any existing data. You typically use this method to -compare existing data to historical data from a snapshot. +If you want to avoid deleting or overwriting existing data, you can instead +rename the indices and data streams you restore. You typically use this method +to compare existing data to historical data from a snapshot. Before you start, ensure the cluster has enough capacity for both the existing and restored data. From 7a0ef8f0b096339ec25c1bcac68f5f394b8d7eab Mon Sep 17 00:00:00 2001 From: James Rodewig <40268737+jrodewig@users.noreply.github.com> Date: Tue, 7 Sep 2021 18:37:36 -0400 Subject: [PATCH 16/40] edits --- .../restore-snapshot.asciidoc | 29 ++++++++++--------- 1 file changed, 15 insertions(+), 14 deletions(-) diff --git a/docs/reference/snapshot-restore/restore-snapshot.asciidoc b/docs/reference/snapshot-restore/restore-snapshot.asciidoc index 0c7ece5436082..83bf880d3a54e 100644 --- a/docs/reference/snapshot-restore/restore-snapshot.asciidoc +++ b/docs/reference/snapshot-restore/restore-snapshot.asciidoc @@ -101,12 +101,12 @@ POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore [[close-restore]] ==== Close and restore -You can only restore an existing index if it's closed. Typically, you'd only use -this method if you've already closed an index for other reasons. Otherwise, the +You can only restore an existing index if it's closed. Typically, you'd use this +method if you've already closed the index for other reasons. Otherwise, the results are the same as the <> method. -You can close an index using the <>. -You can't close a data stream. +You can close an index using the <>. You can't +close a data stream. // Only display warning in 7.x. // In 7.x, `action.destructive_requires_name` defaults to false. @@ -136,8 +136,9 @@ GET _snapshot/my_repository/my_snapshot_2099.05.06?index_details=true&filter_pat ---- // TEST[continued] -If the primary shard counts match, you can restore the closed index. Restoring -the index deletes its existing data and automatically opens it. +If the primary shard counts match, you can restore the closed index. Restoring a +closed index deletes its existing data. The restore operation also automatically +opens the index. [source,console] ---- @@ -291,20 +292,20 @@ DELETE _data_stream/logs-my_app-default [[restore-different-cluster]] === Restore to a different cluster -TIP: {ess} can help you restore snapshots from other deployment. See +TIP: {ess} can help you restore snapshots from other deployments. See {cloud}/ec-restoring-snapshots.html#ec-restore-across-clusters[Restore across clusters]. -The information stored in a snapshot is not tied to a particular cluster or a cluster name. Therefore, it's possible to -restore a snapshot created from one cluster into another cluster by registering the repository that contains the snapshot in the new cluster and starting the restore process. - -The topology of the new cluster does not have to match the cluster where the -snapshot was created. However, the versions must be -<>. +Snapshot aren't tied to a particular cluster or a cluster name. You can create a +snapshot in one cluster and restore it in another cluster with a +<>. The topology of +the clusters don't need to match. If the new cluster has a smaller capacity, consider the following changes: -* Ensure that the new cluster has enough capacity to store all data streams and indices in the snapshot. +* Ensure the new cluster has enough capacity to store any restored data streams +and + indices. * Restore fewer indices and data streams from the snapshot. * Change the index settings during the restore operation to reduce the <>. From d6db55f24806b0c71879493503fd58b93368017a Mon Sep 17 00:00:00 2001 From: James Rodewig <40268737+jrodewig@users.noreply.github.com> Date: Tue, 7 Sep 2021 18:55:46 -0400 Subject: [PATCH 17/40] edits --- .../restore-snapshot.asciidoc | 26 +++++++++---------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/docs/reference/snapshot-restore/restore-snapshot.asciidoc b/docs/reference/snapshot-restore/restore-snapshot.asciidoc index 83bf880d3a54e..e4f0bdff6daf5 100644 --- a/docs/reference/snapshot-restore/restore-snapshot.asciidoc +++ b/docs/reference/snapshot-restore/restore-snapshot.asciidoc @@ -13,7 +13,7 @@ In this guide, you'll learn how to: * Cancel an ongoing restore The guide also provides tips for <> and <> and <>. [discrete] @@ -122,7 +122,7 @@ POST my-index/_close ---- // TEST[setup:setup-snapshots] -To restore the closed index, the index in the snapshot must have the same +To restore a closed index, the index in the snapshot must have the same number of primary shards. You can check the primary shard counts using the <> and <> APIs. @@ -156,7 +156,8 @@ POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore If you want to avoid deleting or overwriting existing data, you can instead rename the indices and data streams you restore. You typically use this method -to compare existing data to historical data from a snapshot. +to compare existing data to historical data from a snapshot. For example, you +can use this method to review documents after an accidental update or deletion. Before you start, ensure the cluster has enough capacity for both the existing and restored data. @@ -217,8 +218,8 @@ POST _reindex [[monitor-restore]] === Monitor a restore -The restore operation uses the <> to restore -an index's primary shards from a snapshot. While the restore operation +The restore operation uses the <> to +restore an index's primary shards from a snapshot. While the restore operation recovers primary shards, the cluster will have a `yellow` <>. @@ -301,13 +302,12 @@ snapshot in one cluster and restore it in another cluster with a <>. The topology of the clusters don't need to match. -If the new cluster has a smaller capacity, consider the following changes: +You should ensure the new cluster has enough capacity for any data streams or +indices you want to restore from the snapshot. If the new cluster has a smaller +capacity, consider: -* Ensure the new cluster has enough capacity to store any restored data streams -and - indices. -* Restore fewer indices and data streams from the snapshot. -* Change the index settings during the restore operation to reduce the +* Restoring fewer indices and data streams. +* Changing the index settings during the restore operation to reduce the <>. + For example, the following restore snapshot API request uses the @@ -336,8 +336,8 @@ restoring incompatible settings. If you need to restore a snapshot with incompat the <>. [discrete] -[[troubleshoot-snapshot-restore]] -=== Troubleshoot snapshot restore errors +[[troubleshoot-restore]] +=== Troubleshoot restore errors Here's how to resolve common errors returned by restore requests. From 23b83c9b95efca1c0741e79ac22ed69abf60bc58 Mon Sep 17 00:00:00 2001 From: James Rodewig <40268737+jrodewig@users.noreply.github.com> Date: Tue, 7 Sep 2021 19:43:22 -0400 Subject: [PATCH 18/40] fix typos --- docs/reference/snapshot-restore/restore-snapshot.asciidoc | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/reference/snapshot-restore/restore-snapshot.asciidoc b/docs/reference/snapshot-restore/restore-snapshot.asciidoc index e4f0bdff6daf5..70fcf205d677b 100644 --- a/docs/reference/snapshot-restore/restore-snapshot.asciidoc +++ b/docs/reference/snapshot-restore/restore-snapshot.asciidoc @@ -297,10 +297,10 @@ TIP: {ess} can help you restore snapshots from other deployments. See {cloud}/ec-restoring-snapshots.html#ec-restore-across-clusters[Restore across clusters]. -Snapshot aren't tied to a particular cluster or a cluster name. You can create a -snapshot in one cluster and restore it in another cluster with a +Snapshots aren't tied to a particular cluster or a cluster name. You can create +a snapshot in one cluster and restore it in another cluster with a <>. The topology of -the clusters don't need to match. +the clusters doesn't need to match. You should ensure the new cluster has enough capacity for any data streams or indices you want to restore from the snapshot. If the new cluster has a smaller From 9d8ebf165d64757d5d8d1acc71ed9b158b3fe807 Mon Sep 17 00:00:00 2001 From: James Rodewig <40268737+jrodewig@users.noreply.github.com> Date: Tue, 7 Sep 2021 19:44:31 -0400 Subject: [PATCH 19/40] edit --- docs/reference/snapshot-restore/restore-snapshot.asciidoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/reference/snapshot-restore/restore-snapshot.asciidoc b/docs/reference/snapshot-restore/restore-snapshot.asciidoc index 70fcf205d677b..df14302b20f99 100644 --- a/docs/reference/snapshot-restore/restore-snapshot.asciidoc +++ b/docs/reference/snapshot-restore/restore-snapshot.asciidoc @@ -298,8 +298,8 @@ TIP: {ess} can help you restore snapshots from other deployments. See clusters]. Snapshots aren't tied to a particular cluster or a cluster name. You can create -a snapshot in one cluster and restore it in another cluster with a -<>. The topology of +a snapshot in one cluster and restore it in another +<>. The topology of the clusters doesn't need to match. You should ensure the new cluster has enough capacity for any data streams or From 0722896160eac5fd1290534807c825d86b95c253 Mon Sep 17 00:00:00 2001 From: James Rodewig <40268737+jrodewig@users.noreply.github.com> Date: Tue, 7 Sep 2021 21:15:33 -0400 Subject: [PATCH 20/40] edit --- .../snapshot-restore/restore-snapshot.asciidoc | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/docs/reference/snapshot-restore/restore-snapshot.asciidoc b/docs/reference/snapshot-restore/restore-snapshot.asciidoc index df14302b20f99..03289f71ae8a2 100644 --- a/docs/reference/snapshot-restore/restore-snapshot.asciidoc +++ b/docs/reference/snapshot-restore/restore-snapshot.asciidoc @@ -102,7 +102,7 @@ POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore ==== Close and restore You can only restore an existing index if it's closed. Typically, you'd use this -method if you've already closed the index for other reasons. Otherwise, the +method if you've already closed an index for other reasons. Otherwise, the results are the same as the <> method. You can close an index using the <>. You can't @@ -302,13 +302,14 @@ a snapshot in one cluster and restore it in another <>. The topology of the clusters doesn't need to match. -You should ensure the new cluster has enough capacity for any data streams or -indices you want to restore from the snapshot. If the new cluster has a smaller -capacity, consider: +Before you start a restore operation, ensure the new cluster has enough capacity +for any data streams or indices you want to restore. If the new cluster has a +smaller capacity, you can: -* Restoring fewer indices and data streams. -* Changing the index settings during the restore operation to reduce the -<>. +* Add nodes or upgrade your hardware to increase capacity. +* Restore fewer indices and data streams. +* Reduce the <> for +restored indices. + For example, the following restore snapshot API request uses the `index_settings` option to set `index.number_of_replicas` to `1`. From ab6920b459df1414a01990659adf0e8ece55cd49 Mon Sep 17 00:00:00 2001 From: James Rodewig <40268737+jrodewig@users.noreply.github.com> Date: Wed, 8 Sep 2021 09:59:31 -0400 Subject: [PATCH 21/40] Add prereq for enterprise search --- .../snapshot-restore/apis/restore-snapshot-api.asciidoc | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/reference/snapshot-restore/apis/restore-snapshot-api.asciidoc b/docs/reference/snapshot-restore/apis/restore-snapshot-api.asciidoc index 98c39b8e3bd0c..7344da7a84821 100644 --- a/docs/reference/snapshot-restore/apis/restore-snapshot-api.asciidoc +++ b/docs/reference/snapshot-restore/apis/restore-snapshot-api.asciidoc @@ -79,6 +79,10 @@ POST /_snapshot/my_repository/my_snapshot/_restore <> with data stream enabled. Without a matching index template, a data stream can't roll over or create backing indices. + +* If your snapshot contains data from App Search or Workplace Search, ensure +you've restored the {enterprise-search-ref}/encryption-keys.html[Enterprise +Search encryption key] before restoring the snapshot. // end::restore-prereqs[] [[restore-snapshot-api-path-params]] From d75398d19fd52fdf8a50e3808b7c3289474bf48f Mon Sep 17 00:00:00 2001 From: James Rodewig <40268737+jrodewig@users.noreply.github.com> Date: Tue, 14 Sep 2021 09:41:25 -0400 Subject: [PATCH 22/40] Add tip for excluding system indices from restore request --- .../snapshot-restore/restore-snapshot.asciidoc | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/docs/reference/snapshot-restore/restore-snapshot.asciidoc b/docs/reference/snapshot-restore/restore-snapshot.asciidoc index 03289f71ae8a2..53f90a4482271 100644 --- a/docs/reference/snapshot-restore/restore-snapshot.asciidoc +++ b/docs/reference/snapshot-restore/restore-snapshot.asciidoc @@ -87,15 +87,27 @@ DELETE my-index # Delete a data stream DELETE _data_stream/logs-my_app-default +---- +// TEST[setup:setup-snapshots] +// TEST[s/_restore/_restore?wait_for_completion=true/] + +You can then restore the index or data stream from the snapshot. + +We recommend you only restore specific indices and data streams. If you use the +<>, specify the `indices` parameter +with a value other than `*`. If you use `*` or the default `indices` value, the +request attempts to restore all indices and data streams, including system +indices. If your cluster contains any existing system indices, the request will +return an error. To restore all indices and data streams except system indices, +use `*,-.*` -# Restore the index and data stream from a snapshot +[source,console] +---- POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore { "indices": "my-index,logs-my_app-default" } ---- -// TEST[setup:setup-snapshots] -// TEST[s/_restore/_restore?wait_for_completion=true/] [discrete] [[close-restore]] From 1e90917cebccc7c73917b66003eda3beebc9eef4 Mon Sep 17 00:00:00 2001 From: James Rodewig <40268737+jrodewig@users.noreply.github.com> Date: Tue, 14 Sep 2021 09:49:34 -0400 Subject: [PATCH 23/40] Fix wrap + missing period --- .../snapshot-restore/restore-snapshot.asciidoc | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/reference/snapshot-restore/restore-snapshot.asciidoc b/docs/reference/snapshot-restore/restore-snapshot.asciidoc index 53f90a4482271..8e072e4b9ffc0 100644 --- a/docs/reference/snapshot-restore/restore-snapshot.asciidoc +++ b/docs/reference/snapshot-restore/restore-snapshot.asciidoc @@ -93,13 +93,13 @@ DELETE _data_stream/logs-my_app-default You can then restore the index or data stream from the snapshot. -We recommend you only restore specific indices and data streams. If you use the -<>, specify the `indices` parameter -with a value other than `*`. If you use `*` or the default `indices` value, the -request attempts to restore all indices and data streams, including system -indices. If your cluster contains any existing system indices, the request will -return an error. To restore all indices and data streams except system indices, -use `*,-.*` +We recommend you only restore specific indices and data streams. If you use +the <>, specify the `indices` +parameter with a value other than `*`. If you use `*` or the default `indices` +value, the request attempts to restore all indices and data streams, including +system indices. If your cluster contains any existing system indices, the +request will return an error. To restore all indices and data streams except +system indices, use `*,-.*`. [source,console] ---- From b00555b83c6079ba8ebf293db3f35dea8f7a8bf5 Mon Sep 17 00:00:00 2001 From: James Rodewig <40268737+jrodewig@users.noreply.github.com> Date: Tue, 14 Sep 2021 09:55:56 -0400 Subject: [PATCH 24/40] Fix snippet tests --- docs/reference/snapshot-restore/restore-snapshot.asciidoc | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/reference/snapshot-restore/restore-snapshot.asciidoc b/docs/reference/snapshot-restore/restore-snapshot.asciidoc index 8e072e4b9ffc0..a65aa95998184 100644 --- a/docs/reference/snapshot-restore/restore-snapshot.asciidoc +++ b/docs/reference/snapshot-restore/restore-snapshot.asciidoc @@ -89,7 +89,7 @@ DELETE my-index DELETE _data_stream/logs-my_app-default ---- // TEST[setup:setup-snapshots] -// TEST[s/_restore/_restore?wait_for_completion=true/] + You can then restore the index or data stream from the snapshot. @@ -108,6 +108,8 @@ POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore "indices": "my-index,logs-my_app-default" } ---- +// TEST[continued] +// TEST[s/_restore/_restore?wait_for_completion=true/] [discrete] [[close-restore]] From cf29921a62e83044dfd4abec2c118d38890994ad Mon Sep 17 00:00:00 2001 From: James Rodewig <40268737+jrodewig@users.noreply.github.com> Date: Tue, 14 Sep 2021 09:57:57 -0400 Subject: [PATCH 25/40] Minor fix --- .../snapshot-restore/restore-snapshot.asciidoc | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/reference/snapshot-restore/restore-snapshot.asciidoc b/docs/reference/snapshot-restore/restore-snapshot.asciidoc index a65aa95998184..5c762b7ba820b 100644 --- a/docs/reference/snapshot-restore/restore-snapshot.asciidoc +++ b/docs/reference/snapshot-restore/restore-snapshot.asciidoc @@ -93,11 +93,11 @@ DELETE _data_stream/logs-my_app-default You can then restore the index or data stream from the snapshot. -We recommend you only restore specific indices and data streams. If you use -the <>, specify the `indices` -parameter with a value other than `*`. If you use `*` or the default `indices` -value, the request attempts to restore all indices and data streams, including -system indices. If your cluster contains any existing system indices, the +We recommend you only restore specific indices and data streams. If you use the +<>, specify the `indices` parameter +with a value other than `*`. If you use `*` or the default `indices` value, the +request attempts to restore all indices and data streams, including system +indices. If your cluster already contains any of these system indices, the request will return an error. To restore all indices and data streams except system indices, use `*,-.*`. From 2134e439d7e07bba6b93a8a14ec8df01a9bcb800 Mon Sep 17 00:00:00 2001 From: James Rodewig <40268737+jrodewig@users.noreply.github.com> Date: Tue, 14 Sep 2021 10:03:15 -0400 Subject: [PATCH 26/40] Clarify --- docs/reference/snapshot-restore/restore-snapshot.asciidoc | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/reference/snapshot-restore/restore-snapshot.asciidoc b/docs/reference/snapshot-restore/restore-snapshot.asciidoc index 5c762b7ba820b..4e34eb35982da 100644 --- a/docs/reference/snapshot-restore/restore-snapshot.asciidoc +++ b/docs/reference/snapshot-restore/restore-snapshot.asciidoc @@ -96,10 +96,10 @@ You can then restore the index or data stream from the snapshot. We recommend you only restore specific indices and data streams. If you use the <>, specify the `indices` parameter with a value other than `*`. If you use `*` or the default `indices` value, the -request attempts to restore all indices and data streams, including system -indices. If your cluster already contains any of these system indices, the -request will return an error. To restore all indices and data streams except -system indices, use `*,-.*`. +request attempts to restore all indices and data streams in the snapshot, +including system indices. If your cluster already contains any of these system +indices, the request will return an error. To restore all indices and data +streams except system indices, use `*,-.*`. [source,console] ---- From 0f343b07b853240e4860db130c2b3caf8daead82 Mon Sep 17 00:00:00 2001 From: James Rodewig <40268737+jrodewig@users.noreply.github.com> Date: Wed, 15 Sep 2021 01:34:36 -0400 Subject: [PATCH 27/40] Rewrite --- .../high-availability/backup-cluster.asciidoc | 6 +- .../restore-cluster-data.asciidoc | 15 - .../apis/restore-snapshot-api.asciidoc | 18 - .../restore-snapshot.asciidoc | 365 +++++++++++++----- 4 files changed, 274 insertions(+), 130 deletions(-) delete mode 100644 docs/reference/high-availability/restore-cluster-data.asciidoc diff --git a/docs/reference/high-availability/backup-cluster.asciidoc b/docs/reference/high-availability/backup-cluster.asciidoc index 3749029355453..73a7d1e4c1c9e 100644 --- a/docs/reference/high-availability/backup-cluster.asciidoc +++ b/docs/reference/high-availability/backup-cluster.asciidoc @@ -9,14 +9,10 @@ To have a complete backup for your cluster: . <> . <> -To restore your cluster from a backup: - -. <> -. <> +To restore your cluster from a backup, see <>. include::backup-cluster-data.asciidoc[] include::backup-cluster-config.asciidoc[] include::backup-and-restore-security-config.asciidoc[] -include::restore-cluster-data.asciidoc[] diff --git a/docs/reference/high-availability/restore-cluster-data.asciidoc b/docs/reference/high-availability/restore-cluster-data.asciidoc deleted file mode 100644 index da58e8c4f3e7f..0000000000000 --- a/docs/reference/high-availability/restore-cluster-data.asciidoc +++ /dev/null @@ -1,15 +0,0 @@ -[[restore-cluster-data]] -=== Restore a cluster's data -++++ -Restore the data -++++ - -include::../snapshot-restore/index.asciidoc[tag=restore-intro] - -[TIP] -==== -If your cluster has {es} {security-features} enabled, the restore API requires the `manage` cluster privilege. There is no bespoke role for the restore process. This privilege is very permissive and should only -be granted to users in the "administrator" category. Specifically, it allows -malicious users to exfiltrate data to a location of their choosing. Automated -tools should not run as users with this privilege. -==== diff --git a/docs/reference/snapshot-restore/apis/restore-snapshot-api.asciidoc b/docs/reference/snapshot-restore/apis/restore-snapshot-api.asciidoc index 7344da7a84821..5b292e09486de 100644 --- a/docs/reference/snapshot-restore/apis/restore-snapshot-api.asciidoc +++ b/docs/reference/snapshot-restore/apis/restore-snapshot-api.asciidoc @@ -182,15 +182,6 @@ backing indices are configured using the data stream's matching index template. (Optional, string) Comma-separated list of indices and data streams to restore. Supports <>. Defaults to all indices and data streams in the snapshot, including system indices. -+ -You can only restore an existing index if it's closed. You can't restore an -existing data stream. If you restore a data stream, you also restore its -backing indices. The restore operation automatically opens restored indices, -including backing indices. -+ -You can use this option to restore only a specific backing index from a data -stream. However, the restore operation doesn't add the restored backing index to -any existing data stream. [[restore-snapshot-api-partial]] `partial`:: @@ -208,15 +199,6 @@ Defines a rename pattern to apply to restored data streams and indices. Data str + The rename pattern is applied as defined by the regular expression that supports referencing the original text, according to the https://docs.oracle.com/javase/8/docs/api/java/util/regex/Matcher.html#appendReplacement-java.lang.StringBuffer-java.lang.String-[`appendReplacement`] logic. -+ -If you rename a data stream, its backing indices are also renamed. For example, -if you rename the `logs-my_app-default` data stream to -`restored-logs-my_app-default`, the backing index -`.ds-logs-my_app-default-2099.03.09-000005` is renamed to -`.ds-restored-logs-my_app-default-2099.03.09-000005`. -+ -If this option produces two or more indices or data streams with the same name, -the restore operation fails. [[restore-snapshot-api-rename-replacement]] `rename_replacement`:: diff --git a/docs/reference/snapshot-restore/restore-snapshot.asciidoc b/docs/reference/snapshot-restore/restore-snapshot.asciidoc index 4e34eb35982da..5625125b2c26b 100644 --- a/docs/reference/snapshot-restore/restore-snapshot.asciidoc +++ b/docs/reference/snapshot-restore/restore-snapshot.asciidoc @@ -8,13 +8,14 @@ indices and data streams after deletion or a hardware failure. In this guide, you'll learn how to: * Retrieve available snapshots -* Restore an index or data stream from a snapshot +* Restore an index or data stream +* Restore an entire cluster * Monitor the restore operation * Cancel an ongoing restore -The guide also provides tips for <> and <>. +This guide also provides tips for <> and <>. [discrete] [[restore-snapshot-prereqs]] @@ -22,6 +23,27 @@ restore errors>>. include::apis/restore-snapshot-api.asciidoc[tag=restore-prereqs] +[discrete] +[[restore-snapshot-considerations]] +=== Considerations + +When restoring a snapshot, keep the following in mind: + +* If you restore a data stream, you also restore its backing indices. + +* You can only restore an existing index if it's <> and +the index in the snapshot has the same number of primary shards. + +* You can't restore an existing open index. This includes +backing indices for a data stream. + +* The restore operation automatically opens restored indices, including backing +indices. + +* You can restore only a specific backing index from a data stream. However, the +restore operation doesn't add the restored backing index to any existing data +stream. + [discrete] [[retrieve-snapshot]] === Retrieve a snapshot @@ -51,15 +73,16 @@ GET _snapshot/my_repository/*?verbose=false [[restore-index-data-stream]] === Restore an index or data stream -You can restore an index or data stream from a snapshot using {kib}'s *Snapshot -and Restore* feature or the <>. +You can restore a snapshot using {kib}'s *Snapshot and Restore* feature or the +<>. -You can't restore an existing open index or an existing data stream. To avoid -conflicts with existing indices and data streams, use one of the following -methods: +In most cases, you only need to restore a specific index or data stream from a +snapshot. However, you can't restore an existing open index. + +To avoid conflicts with existing indices and data streams, use one of the +following methods: * <> -* <> * <> [discrete] @@ -67,18 +90,20 @@ methods: ==== Delete and restore The simplest way to avoid conflicts is to delete an existing index or data -stream before restoring it. To prevent the accidental recreation of the index or -data stream, we recommend you temporarily stop all indexing until the restore -operation is complete. - -// Only display warning in 7.x. -// In 7.x, `action.destructive_requires_name` defaults to false. -ifeval::[ "{major-version}" == "7.x" ] -WARNING: Don't use the <> to target the +stream before restoring it: + +. Delete the existing index or data stream. ++ +-- +To prevent the accidental recreation of the index or data stream, we recommend +you temporarily stop all indexing until the restore operation is complete. + +WARNING: If +<> is +`false`, don't use the <> to target the `*` or `.*` wildcard expression. If you use {es}'s security features, this will delete system indices required for authentication. To delete all regular -indices, use `*,-.*`. -endif::[] +indices, use `*,-.*` instead. [source,console] ---- @@ -89,17 +114,18 @@ DELETE my-index DELETE _data_stream/logs-my_app-default ---- // TEST[setup:setup-snapshots] +-- +. Restore the index or data stream from a snapshot. ++ +-- +By default, a restore request attempts to restore all indices and data +streams in the snapshot, including system indices. If your cluster already +contains one or more of these system indices, the request will return an error. -You can then restore the index or data stream from the snapshot. - -We recommend you only restore specific indices and data streams. If you use the -<>, specify the `indices` parameter -with a value other than `*`. If you use `*` or the default `indices` value, the -request attempts to restore all indices and data streams in the snapshot, -including system indices. If your cluster already contains any of these system -indices, the request will return an error. To restore all indices and data -streams except system indices, use `*,-.*`. +To avoid this error, specify the indices and data streams to restore. To +exclude system indices, append the `-.*` wildcard pattern. To restore all +indices and data streams except system indices, use `*,-.*`. [source,console] ---- @@ -110,65 +136,13 @@ POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore ---- // TEST[continued] // TEST[s/_restore/_restore?wait_for_completion=true/] - -[discrete] -[[close-restore]] -==== Close and restore - -You can only restore an existing index if it's closed. Typically, you'd use this -method if you've already closed an index for other reasons. Otherwise, the -results are the same as the <> method. - -You can close an index using the <>. You can't -close a data stream. - -// Only display warning in 7.x. -// In 7.x, `action.destructive_requires_name` defaults to false. -ifeval::["{major-version}" == "7.x"] -WARNING: Don't use the close index API to target the `*` or `.*` wildcard -expression. If you use {es}'s security features, this will close system indices -required for authentication. To close all regular indices, use `*,-.*`. -endif::[] - -[source,console] ----- -POST my-index/_close ----- -// TEST[setup:setup-snapshots] - -To restore a closed index, the index in the snapshot must have the same -number of primary shards. You can check the primary shard counts using the -<> and <> APIs. - -[source,console] ----- -# Get the primary shard count for the closed index -GET /_cat/indices/my-index?v=true&h=index,pri,status,health - -# Get the primary shard count for the index in the snapshot -GET _snapshot/my_repository/my_snapshot_2099.05.06?index_details=true&filter_path=*.index_details.my-index.shard_count ----- -// TEST[continued] - -If the primary shard counts match, you can restore the closed index. Restoring a -closed index deletes its existing data. The restore operation also automatically -opens the index. - -[source,console] ----- -POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore -{ - "indices": "my-index" -} ----- -// TEST[continued] -// TEST[s/_restore/_restore?wait_for_completion=true/] +-- [discrete] [[rename-on-restore]] ==== Rename on restore -If you want to avoid deleting or overwriting existing data, you can instead +If you want to avoid deleting existing data, you can instead rename the indices and data streams you restore. You typically use this method to compare existing data to historical data from a snapshot. For example, you can use this method to review documents after an accidental update or deletion. @@ -191,6 +165,15 @@ POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore // TEST[setup:setup-snapshots] // TEST[s/_restore/_restore?wait_for_completion=true/] +If the rename options produce two or more indices or data streams with the same +name, the restore operation fails. + +If you rename a data stream, its backing indices are also renamed. For example, +if you rename the `logs-my_app-default` data stream to +`restored-logs-my_app-default`, the backing index +`.ds-logs-my_app-default-2099.03.09-000005` is renamed to +`.ds-restored-logs-my_app-default-2099.03.09-000005`. + When the restore operation is complete, you can compare the original and restored data. If you no longer need an original index or data stream, you can delete it and use a <> to rename the restored one. @@ -228,6 +211,206 @@ POST _reindex ---- // TEST[continued] +[discrete] +[[restore-entire-cluster]] +=== Restore an entire cluster + +In some cases, you need to restore an entire cluster, including the cluster +state and all system indices. These cases should be rare, such as in the event +of a catastrophic data center failure. + +Restoring an entire cluster involves deleting important system indices, including +those used for authentication. Consider whether you can restore specific indices +or data streams instead. + +If you're restoring to a different cluster, see <> +before you start. + +. Configure the cluster's nodes. ++ +If you previously <>, you can copy these files to each node's `$ES_PATH_CONF` +directory. If you restore these files, you must repeat this step for each node +in the cluster. In most cases, this requires a <>. + +. Temporarily stop indexing and turn off the following features: ++ +-- +* ILM ++ +[source,console] +---- +POST _ilm/stop +---- + +//// +[source,console] +---- +POST _ilm/start +---- +// TEST[continued] +//// + +* Machine Learning ++ +[source,console] +---- +POST _ml/set_upgrade_mode?enabled=true +---- + +//// +[source,console] +---- +POST _ml/set_upgrade_mode?enabled=false +---- +// TEST[continued] +//// + +//// +[source,console] +---- +PUT _cluster/settings +{ + "persistent" : { + "xpack.monitoring.collection.enabled": true + } +} +---- +// TEST[continued] +//// + +* Watcher ++ +[source,console] +---- +POST _watcher/_stop +---- + +//// +[source,console] +---- +POST _watcher/_start +---- +// TEST[continued] +//// +-- + +. If you use {es} security features, log in to a node host, navigate to {es} +installation directory, and add a user with the `superuser` role to the file +realm using the <> tool. ++ +For example, the following command creates a user named `restore_user`. ++ +[source,sh] +---- +./bin/elasticsearch-users useradd restore_user -p my_password -r superuser +---- ++ +Use this file realm user to authenticate requests until the restore operation is +complete. + +. Use the <> to set +<> to true. +This lets you delete indices and data streams using wildcards. ++ +[source,console] +---- +PUT _cluster/settings +{ + "persistent" : { + "action.destructive_requires_name" : false + } +} +---- +// TEST[setup:setup-snapshots] + +. Delete all existing indices and data streams on the cluster, including all +system indices. If you use {es} security features, use the previously created +file realm user to authenticate the request. ++ +[source,console] +---- +# Delete all indices +DELETE * + +# Delete all data streams +DELETE _data_stream/* +---- +// TEST[continued] + +. Restore the entire snapshot, including the cluster state. This also restores +any system indices in the snapshot. ++ +-- +[source,console] +---- +POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore +{ + "indices": "*", + "include_global_state": true +} +---- +// TEST[continued] +// TEST[s/_restore/_restore?wait_for_completion=true/] + +If you take dedicated snapshots of the `.security` system index, restore it as well. + +[source,console] +---- +POST _snapshot/my_sec_repository/my_sec_snapshot_2099.05.06/_restore +{ + "indices": ".security", + "include_global_state": false +} +---- +// TEST[continued] +// TEST[s/^/DELETE *\nDELETE _data_stream\/*\n/] +// TEST[s/my_sec_repository/my_repository/] +// TEST[s/my_sec_snapshot_2099.05.06/my_snapshot_2099.05.06/] +// TEST[s/_restore/_restore?wait_for_completion=true/] +// TEST[s/".security"/""/] +-- + +. When the restore operation is complete, resume indexing and restart any +features you stopped: ++ +-- +* ILM ++ +[source,console] +---- +POST _ilm/start +---- + +* Machine Learning ++ +[source,console] +---- +POST _ml/set_upgrade_mode?enabled=false +---- + +* Watcher ++ +[source,console] +---- +POST _watcher/_start +---- +-- + +. If wanted, reset or reconfigure the `action.destructive_requires_name` cluster +setting. ++ +[source,console] +---- +PUT _cluster/settings +{ + "persistent": { + "action.destructive_requires_name": null + } +} +---- + [discrete] [[monitor-restore]] === Monitor a restore @@ -359,16 +542,14 @@ Here's how to resolve common errors returned by restore requests. [discrete] ==== Cannot restore index [] because an open index with same name already exists in the cluster -The request is trying to restore an open index or data stream that already -exists in the cluster. To avoid this error, use one of the following methods: - -* <> -* <> -* <> +You can't restore an open index that already exists. To resolve this error, try +one of the methods in <>. [discrete] ==== Cannot restore index [] with [x] shards from a snapshot of index [] with [y] shards -To restore an existing closed index, the index in the snapshot must have the -same number of primary shards. Try <> or <>. +You can only restore an existing index if it's closed and the index in the +snapshot has the same number of primary shards. This error indicates the index +in the snapshot has a different number of primary shards. + +To resolve this error, try one of the methods in <>. From 1a15fd387c428f5d2712e88cca6b47e09ce7f84a Mon Sep 17 00:00:00 2001 From: James Rodewig <40268737+jrodewig@users.noreply.github.com> Date: Wed, 15 Sep 2021 01:42:04 -0400 Subject: [PATCH 28/40] Cleanup --- .../snapshot-restore/restore-snapshot.asciidoc | 15 +++------------ 1 file changed, 3 insertions(+), 12 deletions(-) diff --git a/docs/reference/snapshot-restore/restore-snapshot.asciidoc b/docs/reference/snapshot-restore/restore-snapshot.asciidoc index 5625125b2c26b..5e62de6e890cf 100644 --- a/docs/reference/snapshot-restore/restore-snapshot.asciidoc +++ b/docs/reference/snapshot-restore/restore-snapshot.asciidoc @@ -90,11 +90,8 @@ following methods: ==== Delete and restore The simplest way to avoid conflicts is to delete an existing index or data -stream before restoring it: +stream before restoring it. -. Delete the existing index or data stream. -+ --- To prevent the accidental recreation of the index or data stream, we recommend you temporarily stop all indexing until the restore operation is complete. @@ -114,11 +111,7 @@ DELETE my-index DELETE _data_stream/logs-my_app-default ---- // TEST[setup:setup-snapshots] --- -. Restore the index or data stream from a snapshot. -+ --- By default, a restore request attempts to restore all indices and data streams in the snapshot, including system indices. If your cluster already contains one or more of these system indices, the request will return an error. @@ -136,7 +129,6 @@ POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore ---- // TEST[continued] // TEST[s/_restore/_restore?wait_for_completion=true/] --- [discrete] [[rename-on-restore]] @@ -296,7 +288,7 @@ POST _watcher/_start //// -- -. If you use {es} security features, log in to a node host, navigate to {es} +. If you use {es} security features, log in to a node host, navigate to the {es} installation directory, and add a user with the `superuser` role to the file realm using the <> tool. + @@ -398,8 +390,7 @@ POST _watcher/_start ---- -- -. If wanted, reset or reconfigure the `action.destructive_requires_name` cluster -setting. +. Reset the `action.destructive_requires_name` cluster setting. + [source,console] ---- From 54e8db8c52d98e3869faa848e042220ec9253598 Mon Sep 17 00:00:00 2001 From: James Rodewig <40268737+jrodewig@users.noreply.github.com> Date: Wed, 15 Sep 2021 01:44:30 -0400 Subject: [PATCH 29/40] add redirect --- docs/reference/redirects.asciidoc | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/reference/redirects.asciidoc b/docs/reference/redirects.asciidoc index a87c1547b6227..d4cd7ebd3d214 100644 --- a/docs/reference/redirects.asciidoc +++ b/docs/reference/redirects.asciidoc @@ -3,6 +3,11 @@ The following pages have moved or been deleted. +[role="exclude",id="restore-cluster-data"] +=== Restore a cluster's data + +See <>. + [role="exclude",id="grok-basics"] === Grok basics From e9e67f090b4bc1621b24cb8b17981e36bd171520 Mon Sep 17 00:00:00 2001 From: James Rodewig <40268737+jrodewig@users.noreply.github.com> Date: Wed, 15 Sep 2021 01:46:17 -0400 Subject: [PATCH 30/40] fix whitespace --- docs/reference/snapshot-restore/restore-snapshot.asciidoc | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/docs/reference/snapshot-restore/restore-snapshot.asciidoc b/docs/reference/snapshot-restore/restore-snapshot.asciidoc index 5e62de6e890cf..b9849ced4b45e 100644 --- a/docs/reference/snapshot-restore/restore-snapshot.asciidoc +++ b/docs/reference/snapshot-restore/restore-snapshot.asciidoc @@ -90,10 +90,9 @@ following methods: ==== Delete and restore The simplest way to avoid conflicts is to delete an existing index or data -stream before restoring it. - -To prevent the accidental recreation of the index or data stream, we recommend -you temporarily stop all indexing until the restore operation is complete. +stream before restoring it. To prevent the accidental re-creation of the index +or data stream, we recommend you temporarily stop all indexing until the restore +operation is complete. WARNING: If <> is From e9554aec26b09c0babedaed0eeff4b107e7db41f Mon Sep 17 00:00:00 2001 From: James Rodewig <40268737+jrodewig@users.noreply.github.com> Date: Wed, 15 Sep 2021 01:47:36 -0400 Subject: [PATCH 31/40] format fix --- docs/reference/snapshot-restore/restore-snapshot.asciidoc | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/reference/snapshot-restore/restore-snapshot.asciidoc b/docs/reference/snapshot-restore/restore-snapshot.asciidoc index b9849ced4b45e..7f59b6a2bd798 100644 --- a/docs/reference/snapshot-restore/restore-snapshot.asciidoc +++ b/docs/reference/snapshot-restore/restore-snapshot.asciidoc @@ -302,8 +302,8 @@ Use this file realm user to authenticate requests until the restore operation is complete. . Use the <> to set -<> to true. -This lets you delete indices and data streams using wildcards. +<> to +`true`. This lets you delete indices and data streams using wildcards. + [source,console] ---- @@ -389,7 +389,7 @@ POST _watcher/_start ---- -- -. Reset the `action.destructive_requires_name` cluster setting. +. If wanted, reset the `action.destructive_requires_name` cluster setting. + [source,console] ---- From 4f698d5e72e841a2317c93ee029af8567a10c4c7 Mon Sep 17 00:00:00 2001 From: James Rodewig <40268737+jrodewig@users.noreply.github.com> Date: Wed, 15 Sep 2021 02:09:17 -0400 Subject: [PATCH 32/40] fixes --- .../restore-snapshot.asciidoc | 24 +++++++++---------- 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/reference/snapshot-restore/restore-snapshot.asciidoc b/docs/reference/snapshot-restore/restore-snapshot.asciidoc index 7f59b6a2bd798..50d3338a257c0 100644 --- a/docs/reference/snapshot-restore/restore-snapshot.asciidoc +++ b/docs/reference/snapshot-restore/restore-snapshot.asciidoc @@ -8,7 +8,7 @@ indices and data streams after deletion or a hardware failure. In this guide, you'll learn how to: * Retrieve available snapshots -* Restore an index or data stream +* Restore an index or data stream from a snapshot * Restore an entire cluster * Monitor the restore operation * Cancel an ongoing restore @@ -27,7 +27,7 @@ include::apis/restore-snapshot-api.asciidoc[tag=restore-prereqs] [[restore-snapshot-considerations]] === Considerations -When restoring a snapshot, keep the following in mind: +When restoring data from a snapshot, keep the following in mind: * If you restore a data stream, you also restore its backing indices. @@ -61,7 +61,7 @@ GET _snapshot // TEST[setup:setup-snapshots] Use the <> to retrieve snapshots for a -repository. +specific repository. [source,console] ---- @@ -94,12 +94,12 @@ stream before restoring it. To prevent the accidental re-creation of the index or data stream, we recommend you temporarily stop all indexing until the restore operation is complete. -WARNING: If -<> is -`false`, don't use the <> to target the -`*` or `.*` wildcard expression. If you use {es}'s security features, this will -delete system indices required for authentication. To delete all regular -indices, use `*,-.*` instead. +WARNING: If the +<> cluster +setting is `false`, don't use the <> to +target the `*` or `.*` wildcard expression. If you use {es}'s security features, +this will delete system indices required for authentication. To delete all +regular indices, use `*,-.*` instead. [source,console] ---- @@ -206,9 +206,9 @@ POST _reindex [[restore-entire-cluster]] === Restore an entire cluster -In some cases, you need to restore an entire cluster, including the cluster -state and all system indices. These cases should be rare, such as in the event -of a catastrophic data center failure. +In some cases, you need to restore an entire cluster from a snapshot, including +the cluster state and all system indices. These cases should be rare, such as in +the event of a catastrophic failure. Restoring an entire cluster involves deleting important system indices, including those used for authentication. Consider whether you can restore specific indices From c59c9ab394bb3f83f8fe51184b8699f5dd0f8e86 Mon Sep 17 00:00:00 2001 From: James Rodewig <40268737+jrodewig@users.noreply.github.com> Date: Wed, 15 Sep 2021 02:14:09 -0400 Subject: [PATCH 33/40] whitespace --- docs/reference/snapshot-restore/restore-snapshot.asciidoc | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/reference/snapshot-restore/restore-snapshot.asciidoc b/docs/reference/snapshot-restore/restore-snapshot.asciidoc index 50d3338a257c0..14858966c1df2 100644 --- a/docs/reference/snapshot-restore/restore-snapshot.asciidoc +++ b/docs/reference/snapshot-restore/restore-snapshot.asciidoc @@ -263,7 +263,7 @@ POST _ml/set_upgrade_mode?enabled=false ---- PUT _cluster/settings { - "persistent" : { + "persistent": { "xpack.monitoring.collection.enabled": true } } @@ -309,8 +309,8 @@ complete. ---- PUT _cluster/settings { - "persistent" : { - "action.destructive_requires_name" : false + "persistent": { + "action.destructive_requires_name": false } } ---- From b240f61df25864f590031bd53e9a81e7bcb159e0 Mon Sep 17 00:00:00 2001 From: James Rodewig <40268737+jrodewig@users.noreply.github.com> Date: Wed, 15 Sep 2021 08:07:44 -0400 Subject: [PATCH 34/40] Clarify config restoration. Remove dupe config restore docs --- ...ackup-and-restore-security-config.asciidoc | 112 ------------------ .../restore-snapshot.asciidoc | 21 +++- 2 files changed, 15 insertions(+), 118 deletions(-) diff --git a/docs/reference/high-availability/backup-and-restore-security-config.asciidoc b/docs/reference/high-availability/backup-and-restore-security-config.asciidoc index 525604a77271b..15c08a6c31aaf 100644 --- a/docs/reference/high-availability/backup-and-restore-security-config.asciidoc +++ b/docs/reference/high-availability/backup-and-restore-security-config.asciidoc @@ -162,115 +162,3 @@ of the users in the native realm and revoke API keys. * If there are indications that the snapshot has been tampered with, do not restore it. There is currently no option for the restore process to detect malicious tampering. - -[[restore-security-configuration]] -=== Restore a cluster's security configuration -++++ -Restore the security configuration -++++ - -NOTE: You can restore a snapshot of the `.security` index only if it was -created in a previous minor version in the same major version. The last minor -version of every major release can convert and read formats of the index for -both its major version and the next one. - -When you restore security configuration you have the option of doing a complete -restore of *all* configurations, including non-security ones, or to only restore -the contents of the `.security` index. As described in -<>, the second option comprises only -resource-type configurations. The first option has the advantage of restoring -a cluster to a clearly defined state from a past point in time. The second option -touches only security configuration resources, but it does not completely restore -the {security-features}. - -To restore your security configuration from a backup, first make sure that the -repository holding `.security` snapshots is installed: - -[source,console] --------------------------------------------------- -GET /_snapshot/my_backup --------------------------------------------------- -// TEST[continued] - -[source,console] --------------------------------------------------- -GET /_snapshot/my_backup/snapshot_1 --------------------------------------------------- -// TEST[continued] - -Then log in to one of the node hosts, navigate to {es} installation directory, -and follow these steps: - -. Add a new user with the `superuser` built-in role to the -<>. -+ --- -For example, create a user named `restore_user`: -[source,shell] --------------------------------------------------- -bin/elasticsearch-users useradd restore_user -p password -r superuser --------------------------------------------------- --- - -. Using the previously created user, delete the existing `.security-6` or -`.security-7` index. -+ --- -[source,shell] --------------------------------------------------- -curl -u restore_user -X DELETE "localhost:9200/.security-*" --------------------------------------------------- -// NOTCONSOLE - -WARNING: After this step any authentication that relies on the `.security` -index will not work. This means that all API calls that authenticate with -native or reserved users will fail, as will any user that relies on a native role. -The file realm user we created in the step above will continue to work -because it is not stored in the `.security` index and uses the built-in -`superuser` role. - --- - -. Using the same user, restore the `.security` index from the snapshot. -+ --- -[source,shell] --------------------------------------------------- - curl -u restore_user -X POST "localhost:9200/_snapshot/my_backup/snapshot_1/_restore" -H 'Content-Type: application/json' -d' - { - "indices": ".security-*", - "include_global_state": true <1> - } - ' --------------------------------------------------- -// NOTCONSOLE - -<1> The `include_global_state: true` is mandatory only for a complete restore. -This will restore the global cluster metadata, which contains configuration -information for the complete cluster. If you set this to `false`, it recovers -only the contents of the `.security` index, such as usernames and password -hashes, API keys, application privileges, role and role mapping definitions. --- - -. Optionally, if you need to review and override the settings that were included -in the snapshot (by the `include_global_state` flag), cherry-pick and -<> that you -<> with the -`GET _cluster/settings` API. - -. If you pursue a complete point in time restore of the cluster, you also have -to restore configuration files. Again, this will restore non-security settings as -well. -+ --- -This entails a straight-up filesystem copy of the backed up configuration files, -overwriting the contents of `$ES_PATH_CONF`, and restarting the node. This -needs to be done on *every node*. Depending on the extent of the differences -between your current cluster configuration and the restored configuration, you -may not be able to perform a rolling restart. If you are performing a full -restore of your configuration directory, we recommend a full cluster restart as -the safest option. Alternatively, you may wish to restore your configuration -files to a separate location on disk and use file comparison tools to review -the differences between your existing configuration and the restored -configuration. --- diff --git a/docs/reference/snapshot-restore/restore-snapshot.asciidoc b/docs/reference/snapshot-restore/restore-snapshot.asciidoc index 14858966c1df2..c6e4b4dff7e62 100644 --- a/docs/reference/snapshot-restore/restore-snapshot.asciidoc +++ b/docs/reference/snapshot-restore/restore-snapshot.asciidoc @@ -217,13 +217,22 @@ or data streams instead. If you're restoring to a different cluster, see <> before you start. -. Configure the cluster's nodes. +. If you <>, you can restore them. If you chose this option, repeat the following +steps for each node in the cluster: + -If you previously <>, you can copy these files to each node's `$ES_PATH_CONF` -directory. If you restore these files, you must repeat this step for each node -in the cluster. In most cases, this requires a <>. +-- +.. Make a filesystem copy of the backed-up configuration files for the node. + +.. Add the backed-up files to the node's `$ES_PATH_CONF` directory. Overwrite +any existing files in the directory. + +.. Ensure the node's `elasticsearch.yml` file contains the appropriate node +roles and settings. + +You must repeat these steps on every node in the cluster. This requires a +<>. +-- . Temporarily stop indexing and turn off the following features: + From fc749bcb1f602bcd268dd3c2f49601d76ec20d64 Mon Sep 17 00:00:00 2001 From: James Rodewig <40268737+jrodewig@users.noreply.github.com> Date: Wed, 15 Sep 2021 13:55:22 -0400 Subject: [PATCH 35/40] More config restore cleanup --- .../restore-snapshot.asciidoc | 23 ++++++++----------- 1 file changed, 9 insertions(+), 14 deletions(-) diff --git a/docs/reference/snapshot-restore/restore-snapshot.asciidoc b/docs/reference/snapshot-restore/restore-snapshot.asciidoc index c6e4b4dff7e62..ed8047b401a65 100644 --- a/docs/reference/snapshot-restore/restore-snapshot.asciidoc +++ b/docs/reference/snapshot-restore/restore-snapshot.asciidoc @@ -218,21 +218,16 @@ If you're restoring to a different cluster, see <> before you start. . If you <>, you can restore them. If you chose this option, repeat the following -steps for each node in the cluster: -+ --- -.. Make a filesystem copy of the backed-up configuration files for the node. - -.. Add the backed-up files to the node's `$ES_PATH_CONF` directory. Overwrite -any existing files in the directory. - -.. Ensure the node's `elasticsearch.yml` file contains the appropriate node -roles and settings. - -You must repeat these steps on every node in the cluster. This requires a +files>>, you can restore them to each node. This step is optional and requires a <>. --- ++ +After you shut down a node, copy the backed-up configuration files over to the +node's `$ES_PATH_CONF` directory. Before restarting the node, ensure +`elasticsearch.yml` contains the appropriate node roles, node name, and +other node-specific settings. ++ +If you choose to perform this step, you must repeat this process on each node in +the cluster. . Temporarily stop indexing and turn off the following features: + From a61d13e6ce971d6a181916b536d9690259c4840b Mon Sep 17 00:00:00 2001 From: James Rodewig <40268737+jrodewig@users.noreply.github.com> Date: Thu, 16 Sep 2021 10:15:35 -0400 Subject: [PATCH 36/40] Address review feedback --- .../apis/restore-snapshot-api.asciidoc | 8 ++--- .../restore-snapshot.asciidoc | 32 ++++++++++++------- 2 files changed, 23 insertions(+), 17 deletions(-) diff --git a/docs/reference/snapshot-restore/apis/restore-snapshot-api.asciidoc b/docs/reference/snapshot-restore/apis/restore-snapshot-api.asciidoc index 5b292e09486de..377b4b2321b1a 100644 --- a/docs/reference/snapshot-restore/apis/restore-snapshot-api.asciidoc +++ b/docs/reference/snapshot-restore/apis/restore-snapshot-api.asciidoc @@ -120,9 +120,7 @@ operation initializes. Defaults to `false`. error for any missing index or data stream. Defaults to `false`. `ignore_index_settings`:: -(Optional, array of string) Index settings to not restore from the snapshot. You -can't use this option to ignore -<>. +(Optional, string or array of strings) Index settings to not restore from the snapshot. You can't use this option to ignore <>. + For data streams, this option only applies to restored backing indices. New backing indices are configured using the data stream's matching index template. @@ -179,9 +177,7 @@ For data streams, this option only applies to restored backing indices. New backing indices are configured using the data stream's matching index template. `indices`:: -(Optional, string) Comma-separated list of indices and data streams to restore. -Supports <>. Defaults to all indices and -data streams in the snapshot, including system indices. +(Optional, string or array of strings) Comma-separated list of indices and data streams to restore. Supports <>. Defaults to all indices and data streams in the snapshot, including system indices. [[restore-snapshot-api-partial]] `partial`:: diff --git a/docs/reference/snapshot-restore/restore-snapshot.asciidoc b/docs/reference/snapshot-restore/restore-snapshot.asciidoc index ed8047b401a65..0edde10c2526e 100644 --- a/docs/reference/snapshot-restore/restore-snapshot.asciidoc +++ b/docs/reference/snapshot-restore/restore-snapshot.asciidoc @@ -2,12 +2,13 @@ == Restore a snapshot This guide shows you how to restore a snapshot. Snapshots are a convenient way -to transfer data between clusters. You can also restore a snapshot to recover -indices and data streams after deletion or a hardware failure. +to store a copy of your data outside of a cluster. You can restore a snapshot +to recover indices and data streams after deletion or a hardware failure. You +can also use snapshots to transfer data between clusters. In this guide, you'll learn how to: -* Retrieve available snapshots +* Get a list of available snapshots * Restore an index or data stream from a snapshot * Restore an entire cluster * Monitor the restore operation @@ -45,14 +46,16 @@ restore operation doesn't add the restored backing index to any existing data stream. [discrete] -[[retrieve-snapshot]] -=== Retrieve a snapshot +[[get-snapshot-list]] +=== Get a list of available snapshots To view a list of available snapshots in {kib}, go to the main menu and click *Stack Management > Snapshot and Restore*. -You can also use the <> to fetch a -list of registered snapshot repositories. +You can also use the <> and the +<> to find snapshots that are available to +restore. First, use the get repository API to fetch a list of registered +snapshot repositories. [source,console] ---- @@ -60,8 +63,7 @@ GET _snapshot ---- // TEST[setup:setup-snapshots] -Use the <> to retrieve snapshots for a -specific repository. +Then use the get snapshot API to get a list of snapshots in a specific repository. [source,console] ---- @@ -349,7 +351,8 @@ POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore // TEST[continued] // TEST[s/_restore/_restore?wait_for_completion=true/] -If you take dedicated snapshots of the `.security` system index, restore it as well. +If you take <>, restore it as well. [source,console] ---- @@ -468,7 +471,8 @@ GET _cluster/allocation/explain?filter_path=index,node_allocation_decisions.node === Cancel a restore You can delete an index or data stream to cancel its ongoing restore. This also -deletes any existing data for the index or data stream. +deletes any existing data in the cluster for the index or data stream. The +operations don't affect the snapshot or its data. [source,console] ---- @@ -493,6 +497,12 @@ a snapshot in one cluster and restore it in another <>. The topology of the clusters doesn't need to match. +To restore a snapshot, its repository must be +<> and available to the new cluster. +If the original cluster still has write access to the repository, register the +repository in `readonly` mode. This prevents multiple clusters from writing to +the repository at the same time and corrupting the repository's contents. + Before you start a restore operation, ensure the new cluster has enough capacity for any data streams or indices you want to restore. If the new cluster has a smaller capacity, you can: From 356d4bf1318c0e898e6e7af0a3021b9501e136a1 Mon Sep 17 00:00:00 2001 From: James Rodewig <40268737+jrodewig@users.noreply.github.com> Date: Thu, 16 Sep 2021 10:24:03 -0400 Subject: [PATCH 37/40] Remove security index restore --- .../restore-snapshot.asciidoc | 22 ++----------------- 1 file changed, 2 insertions(+), 20 deletions(-) diff --git a/docs/reference/snapshot-restore/restore-snapshot.asciidoc b/docs/reference/snapshot-restore/restore-snapshot.asciidoc index 0edde10c2526e..b8d7b0bde9f70 100644 --- a/docs/reference/snapshot-restore/restore-snapshot.asciidoc +++ b/docs/reference/snapshot-restore/restore-snapshot.asciidoc @@ -63,7 +63,8 @@ GET _snapshot ---- // TEST[setup:setup-snapshots] -Then use the get snapshot API to get a list of snapshots in a specific repository. +Then use the get snapshot API to get a list of snapshots in a specific +repository. [source,console] ---- @@ -351,25 +352,6 @@ POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore // TEST[continued] // TEST[s/_restore/_restore?wait_for_completion=true/] -If you take <>, restore it as well. - -[source,console] ----- -POST _snapshot/my_sec_repository/my_sec_snapshot_2099.05.06/_restore -{ - "indices": ".security", - "include_global_state": false -} ----- -// TEST[continued] -// TEST[s/^/DELETE *\nDELETE _data_stream\/*\n/] -// TEST[s/my_sec_repository/my_repository/] -// TEST[s/my_sec_snapshot_2099.05.06/my_snapshot_2099.05.06/] -// TEST[s/_restore/_restore?wait_for_completion=true/] -// TEST[s/".security"/""/] --- - . When the restore operation is complete, resume indexing and restart any features you stopped: + From aa2050937b6226f5e6821ae3d57f248628a60241 Mon Sep 17 00:00:00 2001 From: James Rodewig <40268737+jrodewig@users.noreply.github.com> Date: Thu, 16 Sep 2021 10:33:49 -0400 Subject: [PATCH 38/40] Fix docs build --- docs/reference/snapshot-restore/restore-snapshot.asciidoc | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/reference/snapshot-restore/restore-snapshot.asciidoc b/docs/reference/snapshot-restore/restore-snapshot.asciidoc index b8d7b0bde9f70..45aeec615d4ed 100644 --- a/docs/reference/snapshot-restore/restore-snapshot.asciidoc +++ b/docs/reference/snapshot-restore/restore-snapshot.asciidoc @@ -340,7 +340,6 @@ DELETE _data_stream/* . Restore the entire snapshot, including the cluster state. This also restores any system indices in the snapshot. + --- [source,console] ---- POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore From 89cd5121ff8e5d935fd0e87bfe2545ca2aabc9bb Mon Sep 17 00:00:00 2001 From: James Rodewig <40268737+jrodewig@users.noreply.github.com> Date: Thu, 16 Sep 2021 10:54:22 -0400 Subject: [PATCH 39/40] Fix typo. Remove dupe sentence. --- docs/reference/snapshot-restore/restore-snapshot.asciidoc | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/docs/reference/snapshot-restore/restore-snapshot.asciidoc b/docs/reference/snapshot-restore/restore-snapshot.asciidoc index 45aeec615d4ed..3c2a4720a5765 100644 --- a/docs/reference/snapshot-restore/restore-snapshot.asciidoc +++ b/docs/reference/snapshot-restore/restore-snapshot.asciidoc @@ -310,7 +310,7 @@ complete. . Use the <> to set <> to -`true`. This lets you delete indices and data streams using wildcards. +`false`. This lets you delete indices and data streams using wildcards. + [source,console] ---- @@ -324,8 +324,7 @@ PUT _cluster/settings // TEST[setup:setup-snapshots] . Delete all existing indices and data streams on the cluster, including all -system indices. If you use {es} security features, use the previously created -file realm user to authenticate the request. +system indices. + [source,console] ---- From 90dd48026e3b82478a8ecdb49903742ac318e0de Mon Sep 17 00:00:00 2001 From: James Rodewig <40268737+jrodewig@users.noreply.github.com> Date: Thu, 16 Sep 2021 12:09:30 -0400 Subject: [PATCH 40/40] Clarify cancel effect on snapshot --- docs/reference/snapshot-restore/restore-snapshot.asciidoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/reference/snapshot-restore/restore-snapshot.asciidoc b/docs/reference/snapshot-restore/restore-snapshot.asciidoc index 3c2a4720a5765..26744fdbaeccb 100644 --- a/docs/reference/snapshot-restore/restore-snapshot.asciidoc +++ b/docs/reference/snapshot-restore/restore-snapshot.asciidoc @@ -451,8 +451,8 @@ GET _cluster/allocation/explain?filter_path=index,node_allocation_decisions.node === Cancel a restore You can delete an index or data stream to cancel its ongoing restore. This also -deletes any existing data in the cluster for the index or data stream. The -operations don't affect the snapshot or its data. +deletes any existing data in the cluster for the index or data stream. Deleting +an index or data stream doesn't affect the snapshot or its data. [source,console] ----