Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOCS] Add supported ESS settings to ES docs #57953

Merged
merged 34 commits into from
Jul 2, 2020
Merged

[DOCS] Add supported ESS settings to ES docs #57953

Show file tree
Hide file tree
Changes from 18 commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
e618f56
Adding ESS icons to supported ES settings.
Jun 8, 2020
a51aeff
Adding new file for supported ESS settings.
Jun 8, 2020
5c341a4
Adding supported ESS settings for HTTP and disk-based shard allocation.
Jun 9, 2020
0fe6c20
Adding more supported settings for ESS.
Jun 9, 2020
d3a31bb
Adding descriptions for each Cloud section, plus additional settings.
Jun 10, 2020
a9acebe
Adding new warehouse file for Cloud, plus additional settings.
Jun 11, 2020
ca25a8a
Adding node settings for Cloud.
Jun 12, 2020
d228d5f
Adding audit settings for Cloud.
Jun 12, 2020
9b35c5e
Merge branch 'master' into docs__add-ess-icon-support
Jun 12, 2020
735a092
Resolving merge conflict.
Jun 12, 2020
a74b072
Adding SAML settings (part 1).
Jun 16, 2020
b40d8c1
Adding SAML realm encryption and signing settings.
Jun 17, 2020
ef0a1f7
Adding SAML SSL settings.
Jun 18, 2020
87dfb98
Adding Kerberos realm settings.
Jun 18, 2020
5758c7d
Adding OpenID Connect Realm settings.
Jun 18, 2020
6043cbd
Adding OpenID Connect SSL settings.
Jun 23, 2020
6c66d47
Resolving leftover Git merge markers.
Jun 24, 2020
07b4acc
Removing Cloud settings page and link to it.
Jun 26, 2020
64bead3
Add link to mapping source
Jun 28, 2020
3836137
Update docs/reference/docs/reindex.asciidoc
Jun 28, 2020
3c0d2ac
Incorporate edit of HTTP settings
Jun 28, 2020
d7b9687
Remove "cloud" from tag and ID
Jun 28, 2020
4edea90
Remove "cloud" from tag and update description
Jun 28, 2020
0dae12d
Remove "cloud" from tag and ID
Jun 28, 2020
0a1c0c0
Change "whitelists" to "specifies"
Jun 28, 2020
90e46a6
Remove "cloud" from end tag
Jun 28, 2020
6c95628
Removing cloud from IDs and tags.
Jun 28, 2020
77290c5
Merge branch 'master' into docs__add-ess-icon-support
Jun 30, 2020
4781d1e
Changing link reference to fix build issue.
Jun 30, 2020
9fd3641
Adding index management page for missing settings.
Jun 30, 2020
66a14b7
Removing warehouse file for Cloud and moving settings elsewhere.
Jun 30, 2020
12d1730
Clarifying true/false usage of http.detailed_errors.enabled.
Jun 30, 2020
79042d7
Merge branch 'master' into docs__add-ess-icon-support
Jul 2, 2020
44de9e6
Changing underscore to dash in link to fix ci build.
Jul 2, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
90 changes: 48 additions & 42 deletions docs/reference/docs/reindex.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
Copies documents from a _source_ to a _destination_.

The source and destination can be any pre-existing index, index alias, or
<<data-streams,data stream>>. However, the source and destination must be
<<data-streams,data stream>>. However, the source and destination must be
different. For example, you cannot reindex a data stream into itself.

[IMPORTANT]
Expand All @@ -16,7 +16,8 @@ Reindex requires <<mapping-source-field,`_source`>> to be enabled for
all documents in the source.

The destination must exist and should be configured as wanted before calling `_reindex`.
Reindex does not copy the settings from the source or its associated template.
Reindex does not copy the settings from the source or its associated template.

Mappings, shard counts, replicas, and so on must be configured ahead of time.
=================================================

Expand Down Expand Up @@ -70,8 +71,13 @@ POST _reindex
[[docs-reindex-api-desc]]
==== {api-description-title}

Extracts the <<mapping-source-field,document source>> from the reindex request's source and indexes the documents into the destination.
You can copy all documents to the destination, or reindex a subset of the documents.
// tag::docs-reindex-api-desc-tag[]
Extracts the document source from the source index and indexes the documents into the destination index.
lockewritesdocs marked this conversation as resolved.
Show resolved Hide resolved
You can copy all documents to the destination index, or reindex a subset of the documents.
// end::docs-reindex-api-desc-tag[]

Extracts the <<mapping-source-field,document source>> from the reindex request's source and indexes the documents into the destination.
You can copy all documents to the destination, or reindex a subset of the documents.
lockewritesdocs marked this conversation as resolved.
Show resolved Hide resolved

Just like <<docs-update-by-query,`_update_by_query`>>, `_reindex` gets a
snapshot of the source but its destination must be **different** so
Expand All @@ -95,8 +101,8 @@ any reindex request to a destination data stream must have an `op_type`
of`create`. A reindex can only add new documents to a destination data stream.
It cannot update existing documents in a destination data stream.

By default, version conflicts abort the `_reindex` process.
To continue reindexing if there are conflicts, set the `"conflicts"` request body parameter to `proceed`.
By default, version conflicts abort the `_reindex` process.
To continue reindexing if there are conflicts, set the `"conflicts"` request body parameter to `proceed`.
In this case, the response includes a count of the version conflicts that were encountered.
Note that the handling of other error types is unaffected by the `"conflicts"` parameter.

Expand All @@ -105,9 +111,9 @@ Note that the handling of other error types is unaffected by the `"conflicts"` p

If the request contains `wait_for_completion=false`, {es}
performs some preflight checks, launches the request, and returns a
<<tasks,`task`>> you can use to cancel or get the status of the task.
{es} creates a record of this task as a document at `.tasks/task/${taskId}`.
When you are done with a task, you should delete the task document so
<<tasks,`task`>> you can use to cancel or get the status of the task.
{es} creates a record of this task as a document at `.tasks/task/${taskId}`.
When you are done with a task, you should delete the task document so
{es} can reclaim the space.

[[docs-reindex-from-multiple-sources]]
Expand Down Expand Up @@ -141,7 +147,7 @@ done

Set `requests_per_second` to any positive decimal number (`1.4`, `6`,
`1000`, etc.) to throttle the rate at which `_reindex` issues batches of index
operations. Requests are throttled by padding each batch with a wait time.
operations. Requests are throttled by padding each batch with a wait time.
To disable throttling, set `requests_per_second` to `-1`.

The throttling is done by waiting between batches so that the `scroll` that `_reindex`
Expand All @@ -156,7 +162,7 @@ target_time = 1000 / 500 per second = 2 seconds
wait_time = target_time - write_time = 2 seconds - .5 seconds = 1.5 seconds
--------------------------------------------------

Since the batch is issued as a single `_bulk` request, large batch sizes
Since the batch is issued as a single `_bulk` request, large batch sizes
cause Elasticsearch to create many requests and then wait for a while before
starting the next set. This is "bursty" instead of "smooth".

Expand Down Expand Up @@ -447,13 +453,13 @@ include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=max_docs]
==== {api-request-body-title}

`conflicts`::
(Optional, enum) Set to `proceed` to continue reindexing even if there are conflicts.
(Optional, enum) Set to `proceed` to continue reindexing even if there are conflicts.
Defaults to `abort`.

`source`::
`index`:::
(Required, string) The name of the data stream, index, or index alias you are copying _from_.
Also accepts a comma-separated list to reindex from multiple sources.
(Required, string) The name of the data stream, index, or index alias you are copying _from_.
Also accepts a comma-separated list to reindex from multiple sources.

`max_docs`:::
(Optional, integer) The maximum number of documents to reindex.
Expand All @@ -468,22 +474,22 @@ Required when indexing from remote.
`username`::::
(Optional, string) The username to use for authentication with the remote host.
`password`::::
(Optional, string) The password to use for authentication with the remote host.
`socket_timeout`::::
(Optional, string) The password to use for authentication with the remote host.
`socket_timeout`::::
(Optional, <<time-units, time units>>) The remote socket read timeout. Defaults to 30 seconds.
`connect_timeout`::::
`connect_timeout`::::
(Optional, <<time-units, time units>>) The remote connection timeout. Defaults to 30 seconds.

`size`:::
{Optional, integer) The number of documents to index per batch.
Use when indexing from remote to ensure that the batches fit within the on-heap buffer,
which defaults to a maximum size of 100 MB.
{Optional, integer) The number of documents to index per batch.
Use when indexing from remote to ensure that the batches fit within the on-heap buffer,
which defaults to a maximum size of 100 MB.

`slice`:::
`id`::::
(Optional, integer) Slice ID for <<docs-reindex-manual-slice, manual slicing>>.
(Optional, integer) Slice ID for <<docs-reindex-manual-slice, manual slicing>>.
`max`::::
(Optional, integer) Total number of slices.
(Optional, integer) Total number of slices.

`sort`:::
+
Expand All @@ -495,31 +501,31 @@ deprecated::[7.6, Sort in reindex is deprecated. Sorting in reindex was never gu
--

`_source`:::
(Optional, string) If `true` reindexes all source fields.
Set to a list to reindex select fields.
Defaults to `true`.
(Optional, string) If `true` reindexes all source fields.
Set to a list to reindex select fields.
Defaults to `true`.

`dest`::
`index`:::
(Required, string) The name of the data stream, index, or index alias you are copying _to_.

`version_type`:::
(Optional, enum) The versioning to use for the indexing operation.
Valid values: `internal`, `external`, `external_gt`, `external_gte`.
(Optional, enum) The versioning to use for the indexing operation.
Valid values: `internal`, `external`, `external_gt`, `external_gte`.
See <<index-version-types>> for more information.

`op_type`:::
(Optional, enum) Set to create to only index documents that do not already exist (put if absent).
`op_type`:::
(Optional, enum) Set to create to only index documents that do not already exist (put if absent).
Valid values: `index`, `create`. Defaults to `index`.
+
IMPORTANT: To reindex to a data stream destination, this argument must be
`create`.

`script`::
`source`:::
(Optional, string) The script to run to update the document source or metadata when reindexing.
`source`:::
(Optional, string) The script to run to update the document source or metadata when reindexing.
`lang`:::
(Optional, enum) The script language: `painless`, `expression`, `mustache`, `java`.
(Optional, enum) The script language: `painless`, `expression`, `mustache`, `java`.
For more information, see <<modules-scripting>>.


Expand Down Expand Up @@ -623,7 +629,7 @@ POST _reindex
[[docs-reindex-select-max-docs]]
===== Reindex select documents with `max_docs`

You can limit the number of processed documents by setting `max_docs`.
You can limit the number of processed documents by setting `max_docs`.
For example, this request copies a single document from `twitter` to
`new_twitter`:

Expand All @@ -645,7 +651,7 @@ POST _reindex
[[docs-reindex-multiple-sources]]
===== Reindex from multiple sources

The `index` attribute in `source` can be a list, allowing you to copy from lots
The `index` attribute in `source` can be a list, allowing you to copy from lots
of sources in one request. This will copy documents from the
`twitter` and `blog` indices:

Expand Down Expand Up @@ -1032,7 +1038,7 @@ It is not possible to configure SSL in the body of the `_reindex` request.
The following settings are supported:

`reindex.ssl.certificate_authorities`::
List of paths to PEM encoded certificate files that should be trusted.
List of paths to PEM encoded certificate files that should be trusted.
You cannot specify both `reindex.ssl.certificate_authorities` and
`reindex.ssl.truststore.path`.

Expand All @@ -1057,7 +1063,7 @@ or "pkcs12", this setting defaults to `PKCS12`. Otherwise, it defaults to `jks`.

`reindex.ssl.verification_mode`::
Indicates the type of verification to protect against man in the middle attacks
and certificate forgery.
and certificate forgery.
One of `full` (verify the hostname and the certificate path), `certificate`
(verify the certificate path, but not the hostname) or `none` (perform no
verification - this is strongly discouraged in production environments).
Expand All @@ -1077,12 +1083,12 @@ You cannot specify both `reindex.ssl.key` and `reindex.ssl.keystore.path`.
`reindex.ssl.key_passphrase`::
Specifies the passphrase to decrypt the PEM encoded private key
(`reindex.ssl.key`) if it is encrypted.
Cannot be used with `reindex.ssl.secure_key_passphrase`.
Cannot be used with `reindex.ssl.secure_key_passphrase`.

`reindex.ssl.secure_key_passphrase` (<<secure-settings,Secure>>)::
Specifies the passphrase to decrypt the PEM encoded private key
(`reindex.ssl.key`) if it is encrypted.
Cannot be used with `reindex.ssl.key_passphrase`.
Cannot be used with `reindex.ssl.key_passphrase`.

`reindex.ssl.keystore.path`::
Specifies the path to the keystore that contains a private key and certificate
Expand All @@ -1092,11 +1098,11 @@ You cannot specify both `reindex.ssl.key` and `reindex.ssl.keystore.path`.

`reindex.ssl.keystore.type`::
The type of the keystore (`reindex.ssl.keystore.path`). Must be either `jks` or `PKCS12`.
If the keystore path ends in ".p12", ".pfx" or "pkcs12", this setting defaults
If the keystore path ends in ".p12", ".pfx" or "pkcs12", this setting defaults
to `PKCS12`. Otherwise, it defaults to `jks`.

`reindex.ssl.keystore.password`::
The password to the keystore (`reindex.ssl.keystore.path`). This setting cannot be used
The password to the keystore (`reindex.ssl.keystore.path`). This setting cannot be used
with `reindex.ssl.keystore.secure_password`.

`reindex.ssl.keystore.secure_password` (<<secure-settings,Secure>>)::
Expand All @@ -1105,10 +1111,10 @@ This setting cannot be used with `reindex.ssl.keystore.password`.

`reindex.ssl.keystore.key_password`::
The password for the key in the keystore (`reindex.ssl.keystore.path`).
Defaults to the keystore password. This setting cannot be used with
Defaults to the keystore password. This setting cannot be used with
`reindex.ssl.keystore.secure_key_password`.

`reindex.ssl.keystore.secure_key_password` (<<secure-settings,Secure>>)::
The password for the key in the keystore (`reindex.ssl.keystore.path`).
Defaults to the keystore password. This setting cannot be used with
Defaults to the keystore password. This setting cannot be used with
`reindex.ssl.keystore.key_password`.
20 changes: 11 additions & 9 deletions docs/reference/index-modules.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ an index.
[[index-modules-settings]]
== Index Settings

[[index-modules-settings-description]]
// tag::index-modules-settings-description-tag[]
Index level settings can be set per-index. Settings may be:

_static_::
Expand All @@ -23,6 +25,7 @@ _dynamic_::

They can be changed on a live index using the
<<indices-update-settings,update-index-settings>> API.
// end::index-modules-settings-description-tag[]

WARNING: Changing static or dynamic index settings on a closed index could
result in incorrect settings that are impossible to rectify without deleting
Expand All @@ -34,15 +37,14 @@ and recreating the index.
Below is a list of all _static_ index settings that are not associated with any
specific index module:

`index.number_of_shards`::
[[index-number-of-shards]]
// tag::index-number-of-shards-tag[]
`index.number_of_shards` {ess-icon}::
The number of primary shards that an index should have. Defaults to `1`. This setting can only be set at index creation time. It cannot be changed on a closed index.
+
NOTE: The number of shards are limited to `1024` per index. This limitation is a safety limit to prevent accidental creation of indices that can destabilize a cluster due to resource allocation. The limit can be modified by specifying `export ES_JAVA_OPTS="-Des.index.max_number_of_shards=128"` system property on every node that is part of the cluster.

The number of primary shards that an index should have. Defaults to 1.
This setting can only be set at index creation time. It cannot be
changed on a closed index. Note: the number of shards are limited to `1024` per
index. This limitation is a safety limit to prevent accidental creation of indices
that can destabilize a cluster due to resource allocation. The limit can be modified
by specifying `export ES_JAVA_OPTS="-Des.index.max_number_of_shards=128"` system property on every node that is
part of the cluster.
// end::index-number-of-shards-tag[]

`index.shard.check_on_startup`::

Expand Down Expand Up @@ -186,7 +188,7 @@ specific index module:

Deleting documents from an index to release resources - rather than deleting the index itself - can increase the index size over time. When `index.blocks.read_only_allow_delete` is set to `true`, deleting documents is not permitted. However, deleting the index itself releases the read-only index block and makes resources available almost immediately.

IMPORTANT: {es} adds and removes the read-only index block automatically when the disk utilization falls below the high watermark, controlled by <<cluster-routing-flood_stage,cluster.routing.allocation.disk.watermark.flood_stage>>.
IMPORTANT: {es} adds and removes the read-only index block automatically when the disk utilization falls below the high watermark, controlled by <<cluster-routing-flood-stage,cluster.routing.allocation.disk.watermark.flood_stage>>.

`index.blocks.read`::

Expand Down
Loading