From efd8921c140569ac21073fda460081fde49bab48 Mon Sep 17 00:00:00 2001 From: James Rodewig Date: Fri, 13 Sep 2019 09:04:43 -0400 Subject: [PATCH 1/6] [DOCS] Reformat shrink index API docs --- docs/reference/indices/create-index.asciidoc | 2 + docs/reference/indices/shrink-index.asciidoc | 127 ++++++++++++++----- 2 files changed, 99 insertions(+), 30 deletions(-) diff --git a/docs/reference/indices/create-index.asciidoc b/docs/reference/indices/create-index.asciidoc index fef6ff96a5236..afb7ab91232b4 100644 --- a/docs/reference/indices/create-index.asciidoc +++ b/docs/reference/indices/create-index.asciidoc @@ -35,6 +35,7 @@ creating an index, you can specify the following: -- (Optional, string) Name of the index you wish to create. +// tag::index-name-reqs[] Index names must meet the following criteria: - Lowercase only @@ -43,6 +44,7 @@ Index names must meet the following criteria: - Cannot start with `-`, `_`, `+` - Cannot be `.` or `..` - Cannot be longer than 255 bytes (note it is bytes, so multi-byte characters will count towards the 255 limit faster) +// end::index-name-reqs[] -- diff --git a/docs/reference/indices/shrink-index.asciidoc b/docs/reference/indices/shrink-index.asciidoc index 50874fd9eaef4..0dd4b4585a7e1 100644 --- a/docs/reference/indices/shrink-index.asciidoc +++ b/docs/reference/indices/shrink-index.asciidoc @@ -1,32 +1,29 @@ [[indices-shrink-index]] -=== Shrink Index +=== Shrink index API +++++ +Shrink index +++++ -The shrink index API allows you to shrink an existing index into a new index -with fewer primary shards. The requested number of primary shards in the target index -must be a factor of the number of shards in the source index. For example an index with -`8` primary shards can be shrunk into `4`, `2` or `1` primary shards or an index -with `15` primary shards can be shrunk into `5`, `3` or `1`. If the number -of shards in the index is a prime number it can only be shrunk into a single -primary shard. Before shrinking, a (primary or replica) copy of every shard -in the index must be present on the same node. +Shrinks an existing index into a new index with fewer primary shards. -Shrinking works as follows: -* First, it creates a new target index with the same definition as the source - index, but with a smaller number of primary shards. +[source,console] +---- +POST /twitter/_shrink/shrunk-twitter-index +---- +// TEST[s/^/PUT twitter\n{"settings":{"index.number_of_shards":2,"blocks.write":true}}\n/] -* Then it hard-links segments from the source index into the target index. (If - the file system doesn't support hard-linking, then all segments are copied - into the new index, which is a much more time consuming process. Also if using - multiple data paths, shards on different data paths require a full copy of - segment files if they are not on the same disk since hardlinks don’t work across - disks) -* Finally, it recovers the target index as though it were a closed index which - had just been re-opened. +[[shrink-index-api-request]] +==== {api-request-title} + +`POST //_shrink/` -[float] -==== Preparing an index for shrinking +`PUT //_shrink/` + + +[[shrink-index-api-prereqs]] +==== {api-prereq-title} In order to shrink an index, the index must be marked as read-only, and a (primary or replica) copy of every shard in the index must be relocated to the @@ -57,15 +54,44 @@ with the <>, or the <> can be used to wait until all shards have relocated with the `wait_for_no_relocating_shards` parameter. -[float] -==== Shrinking an index + +[[shrink-index-api-desc]] +==== {api-description-title} + +The shrink index API allows you to shrink an existing index into a new index +with fewer primary shards. The requested number of primary shards in the target index +must be a factor of the number of shards in the source index. For example an index with +`8` primary shards can be shrunk into `4`, `2` or `1` primary shards or an index +with `15` primary shards can be shrunk into `5`, `3` or `1`. If the number +of shards in the index is a prime number it can only be shrunk into a single +primary shard. Before shrinking, a (primary or replica) copy of every shard +in the index must be present on the same node. + +Shrinking works as follows: + +* First, it creates a new target index with the same definition as the source + index, but with a smaller number of primary shards. + +* Then it hard-links segments from the source index into the target index. (If + the file system doesn't support hard-linking, then all segments are copied + into the new index, which is a much more time consuming process. Also if using + multiple data paths, shards on different data paths require a full copy of + segment files if they are not on the same disk since hardlinks don’t work across + disks) + +* Finally, it recovers the target index as though it were a closed index which + had just been re-opened. + + +[[shrink-index]] +===== Shrink an index To shrink `my_source_index` into a new index called `my_target_index`, issue the following request: [source,console] -------------------------------------------------- -POST my_source_index/_shrink/my_target_index +POST /my_source_index/_shrink/my_target_index { "settings": { "index.routing.allocation.require._name": null, <1> @@ -108,7 +134,7 @@ and accepts `settings` and `aliases` parameters for the target index: [source,console] -------------------------------------------------- -POST my_source_index/_shrink/my_target_index +POST /my_source_index/_shrink/my_target_index { "settings": { "index.number_of_replicas": 1, @@ -131,8 +157,9 @@ POST my_source_index/_shrink/my_target_index NOTE: Mappings may not be specified in the `_shrink` request. -[float] -==== Monitoring the shrink process + +[[monitor-shrink]] +===== Monitor the shrink process The shrink process can be monitored with the <>, or the <> can be used to wait @@ -150,9 +177,49 @@ shrink process begins. When the shrink operation completes, the shard will become `active`. At that point, Elasticsearch will try to allocate any replicas and may decide to relocate the primary shard to another node. -[float] -==== Wait For Active Shards + +[[shrink-wait-active-shards]] +===== Wait For active shards Because the shrink operation creates a new index to shrink the shards to, the <> setting on index creation applies to the shrink index action as well. + + +[[shrink-index-api-path-params]] +==== {api-path-parms-title} + +``:: +(Required, string) +Name of the source index to shrink. + +``:: ++ +-- +(Required, string) +Name of the target index to create. + +include::{docdir}/indices/create-index.asciidoc[tag=index-name-reqs] +-- + +[[shrink-index-api-query-params]] +==== {api-query-parms-title} + +include::{docdir}/rest-api/common-parms.asciidoc[tag=wait_for_active_shards] + +include::{docdir}/rest-api/common-parms.asciidoc[tag=timeoutparms] + + +[[shrink-index-api-request-body]] +==== {api-request-body-title} + +`aliases`:: +(Optional, <>) +Index aliases which include the target index. +See <>. + +`settings`:: +(Optional, <>) +Configuration options for the target index. +See <>. + From d8789a1d49a547ea4d198dda0d3799b239e62d73 Mon Sep 17 00:00:00 2001 From: James Rodewig Date: Mon, 16 Sep 2019 13:14:26 -0400 Subject: [PATCH 2/6] Add target-index to common-parms --- docs/reference/indices/shrink-index.asciidoc | 9 +-------- docs/reference/rest-api/common-parms.asciidoc | 11 +++++++++++ 2 files changed, 12 insertions(+), 8 deletions(-) diff --git a/docs/reference/indices/shrink-index.asciidoc b/docs/reference/indices/shrink-index.asciidoc index 0dd4b4585a7e1..0bee429be218d 100644 --- a/docs/reference/indices/shrink-index.asciidoc +++ b/docs/reference/indices/shrink-index.asciidoc @@ -193,14 +193,7 @@ on index creation applies to the shrink index action as well. (Required, string) Name of the source index to shrink. -``:: -+ --- -(Required, string) -Name of the target index to create. - -include::{docdir}/indices/create-index.asciidoc[tag=index-name-reqs] --- +include::{docdir}/rest-api/common-parms.asciidoc[tag=target-index] [[shrink-index-api-query-params]] ==== {api-query-parms-title} diff --git a/docs/reference/rest-api/common-parms.asciidoc b/docs/reference/rest-api/common-parms.asciidoc index 3e860e58557c9..f642aea421c39 100644 --- a/docs/reference/rest-api/common-parms.asciidoc +++ b/docs/reference/rest-api/common-parms.asciidoc @@ -506,6 +506,17 @@ tag::stats[] purposes. end::stats[] +tag::target-index[] +``:: ++ +-- +(Required, string) +Name of the target index to create. + +include::{docdir}/indices/create-index.asciidoc[tag=index-name-reqs] +-- +end::target-index[] + tag::terminate_after[] `terminate_after`:: (Optional, integer) The maximum number of documents to collect for each shard, From fc13ae7a2c9cd2165ebf7209929f927e92286551 Mon Sep 17 00:00:00 2001 From: James Rodewig Date: Mon, 16 Sep 2019 13:21:28 -0400 Subject: [PATCH 3/6] Add target-index aliases and settings to common-parms --- docs/reference/indices/shrink-index.asciidoc | 11 ++--------- docs/reference/rest-api/common-parms.asciidoc | 15 +++++++++++++++ 2 files changed, 17 insertions(+), 9 deletions(-) diff --git a/docs/reference/indices/shrink-index.asciidoc b/docs/reference/indices/shrink-index.asciidoc index 0bee429be218d..d0f603836a444 100644 --- a/docs/reference/indices/shrink-index.asciidoc +++ b/docs/reference/indices/shrink-index.asciidoc @@ -206,13 +206,6 @@ include::{docdir}/rest-api/common-parms.asciidoc[tag=timeoutparms] [[shrink-index-api-request-body]] ==== {api-request-body-title} -`aliases`:: -(Optional, <>) -Index aliases which include the target index. -See <>. - -`settings`:: -(Optional, <>) -Configuration options for the target index. -See <>. +include::{docdir}/rest-api/common-parms.asciidoc[tag=target-index-aliases] +include::{docdir}/rest-api/common-parms.asciidoc[tag=target-index-settings] diff --git a/docs/reference/rest-api/common-parms.asciidoc b/docs/reference/rest-api/common-parms.asciidoc index f642aea421c39..f5f3e19789b40 100644 --- a/docs/reference/rest-api/common-parms.asciidoc +++ b/docs/reference/rest-api/common-parms.asciidoc @@ -10,6 +10,14 @@ tag::aliases[] index. See <>. end::aliases[] +tag::target-index-aliases[] +`aliases`:: +(Optional, <>) +Index aliases which include the target index. +See <>. +end::target-index-aliases[] + + tag::allow-no-indices[] `allow_no_indices`:: (Optional, boolean) If `true`, @@ -467,6 +475,13 @@ tag::settings[] options for the index. See <>. end::settings[] +tag::target-index-settings[] +`settings`:: +(Optional, <>) +Configuration options for the target index. +See <>. +end::target-index-settings[] + tag::segment-size[] Disk space used by the segment, such as `50kb`. end::segment-size[] From 5063183d570ecba590875a5c161890c2da239b08 Mon Sep 17 00:00:00 2001 From: James Rodewig Date: Mon, 16 Sep 2019 13:38:35 -0400 Subject: [PATCH 4/6] reword --- docs/reference/indices/shrink-index.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/indices/shrink-index.asciidoc b/docs/reference/indices/shrink-index.asciidoc index d0f603836a444..c0743b83589ed 100644 --- a/docs/reference/indices/shrink-index.asciidoc +++ b/docs/reference/indices/shrink-index.asciidoc @@ -27,7 +27,7 @@ POST /twitter/_shrink/shrunk-twitter-index In order to shrink an index, the index must be marked as read-only, and a (primary or replica) copy of every shard in the index must be relocated to the -same node and have <> `green`. +same node and have a <> status of `green`. These two conditions can be achieved with the following request: From aa96d7f7ca373a0fae1a971a8b26f6a77f586b36 Mon Sep 17 00:00:00 2001 From: James Rodewig Date: Mon, 16 Sep 2019 13:40:24 -0400 Subject: [PATCH 5/6] reword --- docs/reference/indices/shrink-index.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/indices/shrink-index.asciidoc b/docs/reference/indices/shrink-index.asciidoc index c0743b83589ed..a260d52850cd8 100644 --- a/docs/reference/indices/shrink-index.asciidoc +++ b/docs/reference/indices/shrink-index.asciidoc @@ -27,7 +27,7 @@ POST /twitter/_shrink/shrunk-twitter-index In order to shrink an index, the index must be marked as read-only, and a (primary or replica) copy of every shard in the index must be relocated to the -same node and have a <> status of `green`. +same node and have a <> status of `green`. These two conditions can be achieved with the following request: From e6f9b25b73619e534067145545500df7417bd0e2 Mon Sep 17 00:00:00 2001 From: James Rodewig Date: Fri, 4 Oct 2019 10:39:24 -0400 Subject: [PATCH 6/6] iter --- docs/reference/indices/shrink-index.asciidoc | 23 ++++++++++++-------- 1 file changed, 14 insertions(+), 9 deletions(-) diff --git a/docs/reference/indices/shrink-index.asciidoc b/docs/reference/indices/shrink-index.asciidoc index a260d52850cd8..762d7bc0bf253 100644 --- a/docs/reference/indices/shrink-index.asciidoc +++ b/docs/reference/indices/shrink-index.asciidoc @@ -25,11 +25,13 @@ POST /twitter/_shrink/shrunk-twitter-index [[shrink-index-api-prereqs]] ==== {api-prereq-title} -In order to shrink an index, the index must be marked as read-only, and a -(primary or replica) copy of every shard in the index must be relocated to the -same node and have a <> status of `green`. +Before you can shrink an index: -These two conditions can be achieved with the following request: +* The index must be read-only. +* A copy of every shard in the index must reside on the same node. +* The <> status must be green. + +These three conditions can be achieved with the following request: [source,console] -------------------------------------------------- @@ -67,19 +69,22 @@ of shards in the index is a prime number it can only be shrunk into a single primary shard. Before shrinking, a (primary or replica) copy of every shard in the index must be present on the same node. -Shrinking works as follows: +[[how-shrink-works]] +===== How shrinking works + +A shrink operation: -* First, it creates a new target index with the same definition as the source +. Creates a new target index with the same definition as the source index, but with a smaller number of primary shards. -* Then it hard-links segments from the source index into the target index. (If +. Hard-links segments from the source index into the target index. (If the file system doesn't support hard-linking, then all segments are copied into the new index, which is a much more time consuming process. Also if using multiple data paths, shards on different data paths require a full copy of segment files if they are not on the same disk since hardlinks don’t work across disks) -* Finally, it recovers the target index as though it were a closed index which +. Recovers the target index as though it were a closed index which had just been re-opened. @@ -179,7 +184,7 @@ replicas and may decide to relocate the primary shard to another node. [[shrink-wait-active-shards]] -===== Wait For active shards +===== Wait for active shards Because the shrink operation creates a new index to shrink the shards to, the <> setting