From 04214f288d7857e7bf369229c229af5b62e58c24 Mon Sep 17 00:00:00 2001 From: James Rodewig Date: Fri, 13 Sep 2019 11:00:25 -0400 Subject: [PATCH 1/7] [DOCS] Reformat split index API docs --- docs/reference/indices/create-index.asciidoc | 2 + docs/reference/indices/split-index.asciidoc | 147 +++++++++++++------ 2 files changed, 104 insertions(+), 45 deletions(-) diff --git a/docs/reference/indices/create-index.asciidoc b/docs/reference/indices/create-index.asciidoc index fef6ff96a5236..afb7ab91232b4 100644 --- a/docs/reference/indices/create-index.asciidoc +++ b/docs/reference/indices/create-index.asciidoc @@ -35,6 +35,7 @@ creating an index, you can specify the following: -- (Optional, string) Name of the index you wish to create. +// tag::index-name-reqs[] Index names must meet the following criteria: - Lowercase only @@ -43,6 +44,7 @@ Index names must meet the following criteria: - Cannot start with `-`, `_`, `+` - Cannot be `.` or `..` - Cannot be longer than 255 bytes (note it is bytes, so multi-byte characters will count towards the 255 limit faster) +// end::index-name-reqs[] -- diff --git a/docs/reference/indices/split-index.asciidoc b/docs/reference/indices/split-index.asciidoc index 1db5522cd4791..fa7c307c22fb1 100644 --- a/docs/reference/indices/split-index.asciidoc +++ b/docs/reference/indices/split-index.asciidoc @@ -1,6 +1,54 @@ [[indices-split-index]] === Split Index +Splits an existing index into a new index with more primary shards. + +[source,console] +---- +POST /twitter/_split/split-twitter-index +{ + "settings": { + "index.number_of_shards": 2 + } +} +---- +// TEST[s/^/PUT twitter\n{"settings":{"blocks.write":true}}\n/] + + +[[split-index-api-request]] +==== {api-request-title} + +`POST //_shrink/` + +`PUT //_shrink/` + + +[[split-index-api-prereqs]] +==== {api-prereq-title} + +In order to split an index, the index must be marked as read-only, +and have <> `green`. + +This can be achieved with the following request: + +[source,console] +-------------------------------------------------- +PUT /my_source_index/_settings +{ + "settings": { + "index.blocks.write": true <1> + } +} +-------------------------------------------------- +// TEST[s/^/PUT my_source_index\n/] + +<1> Prevents write operations to this index while still allowing metadata + changes like deleting the index. + + +[[split-index-api-desc]] +==== {api-description-title} + The split index API allows you to split an existing index into a new index, where each original primary shard is split into two or more primary shards in the new index. @@ -31,8 +79,9 @@ index may by split into an arbitrary number of shards greater than 1. The properties of the default number of routing shards will then apply to the newly split index. -[float] -==== How does splitting work? + +[[how-split-works]] +===== How does splitting work? Splitting works as follows: @@ -49,9 +98,9 @@ Splitting works as follows: * Finally, it recovers the target index as though it were a closed index which had just been re-opened. -[float] + [[incremental-resharding]] -==== Why doesn't Elasticsearch support incremental resharding? +===== Why doesn't Elasticsearch support incremental resharding? Going from `N` shards to `N+1` shards, aka. incremental resharding, is indeed a feature that is supported by many key-value stores. Adding a new shard and @@ -80,49 +129,16 @@ covers both the old and the new index for read operations. Assuming that the old and new indices have respectively +M+ and +N+ shards, this has no overhead compared to searching an index that would have +M+N+ shards. -[float] -==== Preparing an index for splitting -Create a new index: - -[source,console] --------------------------------------------------- -PUT my_source_index -{ - "settings": { - "index.number_of_shards" : 1 - } -} --------------------------------------------------- - -In order to split an index, the index must be marked as read-only, -and have <> `green`. - -This can be achieved with the following request: - -[source,console] --------------------------------------------------- -PUT /my_source_index/_settings -{ - "settings": { - "index.blocks.write": true <1> - } -} --------------------------------------------------- -// TEST[continued] - -<1> Prevents write operations to this index while still allowing metadata - changes like deleting the index. - -[float] -==== Splitting an index +[[split-index]] +===== Split an index To split `my_source_index` into a new index called `my_target_index`, issue the following request: [source,console] -------------------------------------------------- -POST my_source_index/_split/my_target_index +POST /my_source_index/_split/my_target_index { "settings": { "index.number_of_shards": 2 @@ -156,7 +172,7 @@ and accepts `settings` and `aliases` parameters for the target index: [source,console] -------------------------------------------------- -POST my_source_index/_split/my_target_index +POST /my_source_index/_split/my_target_index { "settings": { "index.number_of_shards": 5 <1> @@ -174,8 +190,9 @@ POST my_source_index/_split/my_target_index NOTE: Mappings may not be specified in the `_split` request. -[float] -==== Monitoring the split process + +[[monitor-split]] +==== Monitor the split process The split process can be monitored with the <>, or the <> can be used to wait @@ -193,9 +210,49 @@ split process begins. When the split operation completes, the shard will become `active`. At that point, Elasticsearch will try to allocate any replicas and may decide to relocate the primary shard to another node. -[float] -==== Wait For Active Shards + +[[split-wait-active-shards]] +==== Wait For active shards Because the split operation creates a new index to split the shards to, the <> setting on index creation applies to the split index action as well. + + +[[split-index-api-path-params]] +==== {api-path-parms-title} + +``:: +(Required, string) +Name of the source index to split. + +``:: ++ +-- +(Required, string) +Name of the target index to create. + +include::{docdir}/indices/create-index.asciidoc[tag=index-name-reqs] +-- + + +[[split-index-api-query-params]] +==== {api-query-parms-title} + +include::{docdir}/rest-api/common-parms.asciidoc[tag=wait_for_active_shards] + +include::{docdir}/rest-api/common-parms.asciidoc[tag=timeoutparms] + + +[[split-index-api-request-body]] +==== {api-request-body-title} + +`aliases`:: +(Optional, <>) +Index aliases which include the target index. +See <>. + +`settings`:: +(Optional, <>) +Configuration options for the target index. +See <>. From 79a02d9498c752a9db47ad1ee30018aa70d55272 Mon Sep 17 00:00:00 2001 From: James Rodewig Date: Mon, 16 Sep 2019 13:03:21 -0400 Subject: [PATCH 2/7] Reword prereqs --- docs/reference/indices/split-index.asciidoc | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/reference/indices/split-index.asciidoc b/docs/reference/indices/split-index.asciidoc index fa7c307c22fb1..74726fe36e6da 100644 --- a/docs/reference/indices/split-index.asciidoc +++ b/docs/reference/indices/split-index.asciidoc @@ -26,10 +26,12 @@ POST /twitter/_split/split-twitter-index [[split-index-api-prereqs]] ==== {api-prereq-title} -In order to split an index, the index must be marked as read-only, +To split an index, +the index must be marked as read-only and have <> `green`. -This can be achieved with the following request: +You can do this +with the following request: [source,console] -------------------------------------------------- From 5c740f0fa658916c7b82c350afcf8f66fe89ab10 Mon Sep 17 00:00:00 2001 From: James Rodewig Date: Mon, 16 Sep 2019 13:12:32 -0400 Subject: [PATCH 3/7] Add target-index to common-parms --- docs/reference/indices/split-index.asciidoc | 9 +-------- docs/reference/rest-api/common-parms.asciidoc | 11 +++++++++++ 2 files changed, 12 insertions(+), 8 deletions(-) diff --git a/docs/reference/indices/split-index.asciidoc b/docs/reference/indices/split-index.asciidoc index 74726fe36e6da..11957f65e2cc6 100644 --- a/docs/reference/indices/split-index.asciidoc +++ b/docs/reference/indices/split-index.asciidoc @@ -228,14 +228,7 @@ on index creation applies to the split index action as well. (Required, string) Name of the source index to split. -``:: -+ --- -(Required, string) -Name of the target index to create. - -include::{docdir}/indices/create-index.asciidoc[tag=index-name-reqs] --- +include::{docdir}/rest-api/common-parms.asciidoc[tag=target-index] [[split-index-api-query-params]] diff --git a/docs/reference/rest-api/common-parms.asciidoc b/docs/reference/rest-api/common-parms.asciidoc index 3e860e58557c9..f642aea421c39 100644 --- a/docs/reference/rest-api/common-parms.asciidoc +++ b/docs/reference/rest-api/common-parms.asciidoc @@ -506,6 +506,17 @@ tag::stats[] purposes. end::stats[] +tag::target-index[] +``:: ++ +-- +(Required, string) +Name of the target index to create. + +include::{docdir}/indices/create-index.asciidoc[tag=index-name-reqs] +-- +end::target-index[] + tag::terminate_after[] `terminate_after`:: (Optional, integer) The maximum number of documents to collect for each shard, From 6297208142c937d99d7224123ca37890d5920f62 Mon Sep 17 00:00:00 2001 From: James Rodewig Date: Mon, 16 Sep 2019 13:22:23 -0400 Subject: [PATCH 4/7] Add target-index aliases and settings to common-parms --- docs/reference/indices/split-index.asciidoc | 12 +++--------- docs/reference/rest-api/common-parms.asciidoc | 14 ++++++++++++++ 2 files changed, 17 insertions(+), 9 deletions(-) diff --git a/docs/reference/indices/split-index.asciidoc b/docs/reference/indices/split-index.asciidoc index 11957f65e2cc6..e00fdce57657d 100644 --- a/docs/reference/indices/split-index.asciidoc +++ b/docs/reference/indices/split-index.asciidoc @@ -242,12 +242,6 @@ include::{docdir}/rest-api/common-parms.asciidoc[tag=timeoutparms] [[split-index-api-request-body]] ==== {api-request-body-title} -`aliases`:: -(Optional, <>) -Index aliases which include the target index. -See <>. - -`settings`:: -(Optional, <>) -Configuration options for the target index. -See <>. +include::{docdir}/rest-api/common-parms.asciidoc[tag=target-index-aliases] + +include::{docdir}/rest-api/common-parms.asciidoc[tag=target-index-settings] diff --git a/docs/reference/rest-api/common-parms.asciidoc b/docs/reference/rest-api/common-parms.asciidoc index f642aea421c39..2277cbbf9f659 100644 --- a/docs/reference/rest-api/common-parms.asciidoc +++ b/docs/reference/rest-api/common-parms.asciidoc @@ -10,6 +10,13 @@ tag::aliases[] index. See <>. end::aliases[] +tag::target-index-aliases[] +`aliases`:: +(Optional, <>) +Index aliases which include the target index. +See <>. +end::target-index-aliases[] + tag::allow-no-indices[] `allow_no_indices`:: (Optional, boolean) If `true`, @@ -467,6 +474,13 @@ tag::settings[] options for the index. See <>. end::settings[] +tag::target-index-settings[] +`settings`:: +(Optional, <>) +Configuration options for the target index. +See <>. +end::target-index-settings[] + tag::segment-size[] Disk space used by the segment, such as `50kb`. end::segment-size[] From f76b5d63205d62b1b1dfcd12bbaf85aa053cd1a4 Mon Sep 17 00:00:00 2001 From: James Rodewig Date: Mon, 16 Sep 2019 13:39:06 -0400 Subject: [PATCH 5/7] reword --- docs/reference/indices/split-index.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/indices/split-index.asciidoc b/docs/reference/indices/split-index.asciidoc index e00fdce57657d..321d0c41174f0 100644 --- a/docs/reference/indices/split-index.asciidoc +++ b/docs/reference/indices/split-index.asciidoc @@ -28,7 +28,7 @@ POST /twitter/_split/split-twitter-index To split an index, the index must be marked as read-only -and have <> `green`. +and have a <> status of `green`. You can do this with the following request: From 761dcbf58b211b58e45cbc56f2dcfbebb4754559 Mon Sep 17 00:00:00 2001 From: James Rodewig Date: Mon, 16 Sep 2019 13:40:55 -0400 Subject: [PATCH 6/7] reword --- docs/reference/indices/split-index.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/indices/split-index.asciidoc b/docs/reference/indices/split-index.asciidoc index 321d0c41174f0..eb235486d778a 100644 --- a/docs/reference/indices/split-index.asciidoc +++ b/docs/reference/indices/split-index.asciidoc @@ -28,7 +28,7 @@ POST /twitter/_split/split-twitter-index To split an index, the index must be marked as read-only -and have a <> status of `green`. +and have a <> status of `green`. You can do this with the following request: From 6f981b96518da9cef9c3df87e5a546303f42b0f6 Mon Sep 17 00:00:00 2001 From: James Rodewig Date: Fri, 4 Oct 2019 10:01:30 -0400 Subject: [PATCH 7/7] iter --- docs/reference/indices/split-index.asciidoc | 24 +++++++++++---------- 1 file changed, 13 insertions(+), 11 deletions(-) diff --git a/docs/reference/indices/split-index.asciidoc b/docs/reference/indices/split-index.asciidoc index 21ad7b270e65c..3e4adb304655b 100644 --- a/docs/reference/indices/split-index.asciidoc +++ b/docs/reference/indices/split-index.asciidoc @@ -29,11 +29,13 @@ POST /twitter/_split/split-twitter-index [[split-index-api-prereqs]] ==== {api-prereq-title} -To split an index, -the index must be marked as read-only -and have a <> status of `green`. -You can do this +Before you can split an index: + +* The index must be read-only. +* The <> status must be green. + +You can do make an index read-only with the following request: [source,console] @@ -86,21 +88,21 @@ newly split index. [[how-split-works]] -===== How does splitting work? +===== How splitting works -Splitting works as follows: +A split operation: -* First, it creates a new target index with the same definition as the source +. Creates a new target index with the same definition as the source index, but with a larger number of primary shards. -* Then it hard-links segments from the source index into the target index. (If +. Hard-links segments from the source index into the target index. (If the file system doesn't support hard-linking, then all segments are copied into the new index, which is a much more time consuming process.) -* Once the low level files are created all documents will be `hashed` again to delete +. Hashes all documents again, after low level files are created, to delete documents that belong to a different shard. -* Finally, it recovers the target index as though it were a closed index which +. Recovers the target index as though it were a closed index which had just been re-opened. @@ -217,7 +219,7 @@ replicas and may decide to relocate the primary shard to another node. [[split-wait-active-shards]] -==== Wait For active shards +==== Wait for active shards Because the split operation creates a new index to split the shards to, the <> setting