From 0024edaf68aa7748bd131ea0ef9c41d52e8e9f9c Mon Sep 17 00:00:00 2001 From: debadair Date: Mon, 27 Jan 2020 19:39:24 -0800 Subject: [PATCH] [DOCS] Split off ILM overview to a separate topic. (#51287) * [DOCS} Split off overview to a separate topic. * [DOCS] Incorporated feedback from @jrodewig. --- docs/reference/glossary.asciidoc | 63 ++++++++++++- .../ilm/getting-started-ilm.asciidoc | 4 +- docs/reference/ilm/ilm-and-snapshots.asciidoc | 2 +- .../ilm/ilm-with-existing-indices.asciidoc | 2 +- docs/reference/ilm/index.asciidoc | 94 ++++++------------- docs/reference/ilm/overview-ilm.asciidoc | 69 ++++++++++++++ .../ilm/set-up-lifecycle-policy.asciidoc | 2 +- .../ilm/using-policies-rollover.asciidoc | 2 +- 8 files changed, 166 insertions(+), 72 deletions(-) create mode 100644 docs/reference/ilm/overview-ilm.asciidoc diff --git a/docs/reference/glossary.asciidoc b/docs/reference/glossary.asciidoc index 008667c3cf3cd..c7ee2eb7146c3 100644 --- a/docs/reference/glossary.asciidoc +++ b/docs/reference/glossary.asciidoc @@ -80,6 +80,30 @@ In most cases, the goal of filtering is to reduce the number of documents that h Follower indices are the target indices for <>. They exist in your local cluster and replicate <>. +[[glossary-force-merge]] force merge :: +// tag::force-merge-def[] +// tag::force-merge-def-short[] +Manually trigger a merge to reduce the number of segments in each shard of an index +and free up the space used by deleted documents. +// end::force-merge-def-short[] +You should not force merge indices that are actively being written to. +Merging is normally performed automatically, but you can use force merge after +<> to reduce the shards in the old index to a single segment. +See the {ref}/indices-forcemerge.html[force merge API]. +// end::force-merge-def[] + +[[glossary-freeze]] freeze :: +// tag::freeze-def[] +// tag::freeze-def-short[] +Make an index read-only and minimize its memory footprint. +// end::freeze-def-short[] +Frozen indices can be searched without incurring the overhead of of re-opening a closed index, +but searches are throttled and might be slower. +You can freeze indices to reduce the overhead of keeping older indices searchable +before you are ready to archive or delete them. +See the {ref}/freeze-index-api.html[freeze API]. +// end::freeze-def[] + [[glossary-id]] id :: The ID of a <> identifies a document. The @@ -114,6 +138,18 @@ See {ref}/indices-add-alias.html[Add index alias]. See <>. -- +[[glossary-index-template]] index template :: ++ +-- +// tag::index-template-def[] +// tag::index-template-def-short[] +Defines settings and mappings to apply to new indexes that match a simple naming pattern, such as _logs-*_. +// end::index-template-def-short[] +An index template can also attach a lifecycle policy to the new index. +Index templates are used to automatically configure indices created during <>. +// end::index-template-def[] +-- + [[glossary-leader-index]] leader index :: Leader indices are the source indices for <>. They exist @@ -206,6 +242,19 @@ By default, each primary shard has one replica, but the number of replicas can be changed dynamically on an existing index. A replica shard will never be started on the same node as its primary shard. +[[glossary-rollover]] rollover :: ++ +-- +// tag::rollover-def[] +// tag::rollover-def-short[] +Redirect an alias to begin writing to a new index when the existing index reaches a certain age, number of docs, or size. +// end::rollover-def-short[] +The new index is automatically configured according to any matching <>. +For example, if you're indexing log data, you might use rollover to create daily or weekly indices. +See the {ref}/indices-rollover-index.html[rollover index API]. +// end::rollover-def[] +-- + [[glossary-routing]] routing :: When you index a document, it is stored on a single @@ -220,7 +269,7 @@ time, or a <> in the <>. [[glossary-shard]] shard :: - +// tag::shard-def[] A shard is a single Lucene instance. It is a low-level “worker” unit which is managed automatically by Elasticsearch. An index is a logical namespace which points to <> and @@ -234,6 +283,18 @@ Elasticsearch distributes shards amongst all <> in the <>, and can move shards automatically from one node to another in the case of node failure, or the addition of new nodes. +// end::shard-def[] +-- + +[[glossary-shrink]] shrink :: +// tag::shrink-def[] +// tag::shrink-def-short[] +Reduce the number of primary shards in an index. +// end::shrink-def-short[] +You can shrink an index to reduce its overhead when the request volume drops. +For example, you might opt to shrink an index once it is no longer the write index. +See the {ref}/indices-shrink-index.html[shrink index API]. +// end::shrink-def[] [[glossary-source_field]] source field :: diff --git a/docs/reference/ilm/getting-started-ilm.asciidoc b/docs/reference/ilm/getting-started-ilm.asciidoc index 9aafb9a44fa16..3b3db76608282 100644 --- a/docs/reference/ilm/getting-started-ilm.asciidoc +++ b/docs/reference/ilm/getting-started-ilm.asciidoc @@ -1,7 +1,7 @@ [role="xpack"] [testenv="basic"] [[getting-started-index-lifecycle-management]] -== Getting started with {ilm} +== Get started with {ilm} Let's jump into {ilm} ({ilm-init}) by working through a hands-on scenario. This section will leverage many new concepts unique to {ilm-init} that @@ -170,7 +170,7 @@ GET datastream-*/_ilm/explain The above request will retrieve {ilm-init} execution information for all our managed indices. - +[[36818c6d9f434d387819c30bd9addb14]] [source,console-result] -------------------------------------------------- { diff --git a/docs/reference/ilm/ilm-and-snapshots.asciidoc b/docs/reference/ilm/ilm-and-snapshots.asciidoc index c71fbe0471da2..5de30f11382b4 100644 --- a/docs/reference/ilm/ilm-and-snapshots.asciidoc +++ b/docs/reference/ilm/ilm-and-snapshots.asciidoc @@ -1,7 +1,7 @@ [role="xpack"] [testenv="basic"] [[index-lifecycle-and-snapshots]] -== Restoring snapshots of managed indices +== Restore a managed index When restoring a snapshot that contains indices managed by Index Lifecycle Management, the lifecycle will automatically continue to execute after the diff --git a/docs/reference/ilm/ilm-with-existing-indices.asciidoc b/docs/reference/ilm/ilm-with-existing-indices.asciidoc index 8dfb122da8f92..57b4fe2bd1aaf 100644 --- a/docs/reference/ilm/ilm-with-existing-indices.asciidoc +++ b/docs/reference/ilm/ilm-with-existing-indices.asciidoc @@ -1,7 +1,7 @@ [role="xpack"] [testenv="basic"] [[ilm-with-existing-indices]] -== Using {ilm-init} with existing indices +== Manage existing indices While it is recommended to use {ilm-init} to manage the index lifecycle from start to finish, it may be useful to use {ilm-init} with existing indices, diff --git a/docs/reference/ilm/index.asciidoc b/docs/reference/ilm/index.asciidoc index c26e698efda25..a398f56d1ce9a 100644 --- a/docs/reference/ilm/index.asciidoc +++ b/docs/reference/ilm/index.asciidoc @@ -1,72 +1,35 @@ [role="xpack"] [testenv="basic"] [[index-lifecycle-management]] -= Managing the index lifecycle += Manage the index lifecycle [partintro] -- - -The <> enable you to -automate how you want to manage your indices over time. Rather than simply -performing management actions on your indices on a set schedule, you can base -actions on other factors such as shard size and performance requirements. - -You control how indices are handled as they age by attaching a -lifecycle policy to the index template used to create them. You can update -the policy to modify the lifecycle of both new and existing indices. - -For time series indices, there are four stages in the index lifecycle: - -* Hot--the index is actively being updated and queried. -* Warm--the index is no longer being updated, but is still being queried. -* Cold--the index is no longer being updated and is seldom queried. The -information still needs to be searchable, but it's okay if those queries are -slower. -* Delete--the index is no longer needed and can safely be deleted. - -The lifecycle policy governs how the index transitions through these stages and -the actions that are performed on the index at each stage. The policy can -specify: - -* The maximum size or age at which you want to roll over to a new index. -* The point at which the index is no longer being updated and the number of -primary shards can be reduced. -* When to force a merge to permanently delete documents marked for deletion. -* The point at which the index can be moved to less performant hardware. -* The point at which the availability is not as critical and the number of -replicas can be reduced. -* When the index can be safely deleted. - -For example, if you are indexing metrics data from a fleet of ATMs into -Elasticsearch, you might define a policy that says: - -. When the index reaches 50GB, roll over to a new index. -. Move the old index into the warm stage, mark it read only, and shrink it down -to a single shard. -. After 7 days, move the index into the cold stage and move it to less expensive -hardware. -. Delete the index once the required 30 day retention period is reached. - -*Snapshot Lifecycle Management* - -ILM itself does allow managing indices, however, managing snapshots for a set of -indices is outside of the scope of an index-level policy. Instead, there are -separate APIs for managing snapshot lifecycles. Please see the -<> -documentation for information about configuring snapshots. - -See <>. - -[IMPORTANT] -=========================== -{ilm} does not support mixed-version cluster usage. Although it -may be possible to create such new policies against -newer-versioned nodes, there is no guarantee they will -work as intended. New policies using new actions that -do not exist in the oldest versioned node will cause errors. -=========================== +You can configure {ilm} ({ilm-init}) policies to automatically manage indices according to +your performance, resiliency, and retention requirements. +For example, you could use {ilm-init} to: + +* Create a new index each day, week, or month and archive previous ones +* Spin up a new index when an index reaches a certain size +* Delete stale indices to enforce data retention standards + +[TIP] +To automatically back up your indices and manage snapshots, +use <>. + +* <> +* <> +* <> +* <> +* <> +* <> +* <> +* <> +* <> +* <> -- +include::overview-ilm.asciidoc[] include::getting-started-ilm.asciidoc[] @@ -74,14 +37,15 @@ include::policy-definitions.asciidoc[] include::set-up-lifecycle-policy.asciidoc[] -include::using-policies-rollover.asciidoc[] - include::update-lifecycle-policy.asciidoc[] include::error-handling.asciidoc[] -include::ilm-and-snapshots.asciidoc[] - include::start-stop-ilm.asciidoc[] +include::using-policies-rollover.asciidoc[] + include::ilm-with-existing-indices.asciidoc[] + +include::ilm-and-snapshots.asciidoc[] + diff --git a/docs/reference/ilm/overview-ilm.asciidoc b/docs/reference/ilm/overview-ilm.asciidoc new file mode 100644 index 0000000000000..af3797587fb69 --- /dev/null +++ b/docs/reference/ilm/overview-ilm.asciidoc @@ -0,0 +1,69 @@ +[role="xpack"] +[testenv="basic"] +[[overview-index-lifecycle-management]] +== Index lifecycle management overview + +++++ +Overview +++++ + +You can create and apply {ilm-cap} ({ilm-init}) policies to automatically manage your indices +according to your performance, resiliency, and retention requirements. +Index lifecycle policies can trigger actions such as: + +* **Rollover** - +include::../glossary.asciidoc[tag=rollover-def-short] +* **Shrink** - +include::../glossary.asciidoc[tag=shrink-def-short] +* **Force merge** - +include::../glossary.asciidoc[tag=force-merge-def-short] +* **Freeze** - +include::../glossary.asciidoc[tag=freeze-def-short] +* **Delete** - Permanently remove an index, including all of its data and metadata. + +Typically, you associate a lifecycle policy with an index template so it is automatically applied +to new indices. +You can also apply a policy manually when you create an index. + +{ilm-init} simplifies managing indices in hot-warm-cold architectures, +which are common when you're working with time-series data such as logs and metrics. + +As an index ages, it moves through four possible phases: + +* Hot--the index is actively being updated and queried. +* Warm--the index is no longer being updated, but is still being queried. +* Cold--the index is no longer being updated and is seldom queried. The +information still needs to be searchable, but it's okay if those queries are +slower. +* Delete--the index is no longer needed and can safely be deleted. + +A lifecycle policy controls how an index moves between phases and +what actions to perform during each phase. You can specify: + +* The maximum size or age at which you want to roll over to a new index. +* The point at which the index is no longer being updated and the number of +primary shards can be reduced. +* When to force a merge to permanently delete documents marked for deletion. +* The point at which the index can be moved to less performant hardware. +* The point at which the availability is not as critical and the number of +replicas can be reduced. +* When the index can be safely deleted. + +For example, if you are indexing metrics data from a fleet of ATMs into +Elasticsearch, you might define a policy that says: + +. When the index reaches 50GB, roll over to a new index. +. Move the old index into the warm stage, mark it read only, and shrink it down +to a single shard. +. After 7 days, move the index into the cold stage and move it to less expensive +hardware. +. Delete the index once the required 30 day retention period is reached. + +[IMPORTANT] +=========================== +To use {ilm-init}, all nodes in a cluster must run the same version. +Although it might be possible to create and apply policies in a mixed-version cluster, +there is no guarantee they will work as intended. +Attempting to use a policy that contains actions that aren't +supported on all nodes in a cluster will cause errors. +=========================== \ No newline at end of file diff --git a/docs/reference/ilm/set-up-lifecycle-policy.asciidoc b/docs/reference/ilm/set-up-lifecycle-policy.asciidoc index 60013951afd89..a8f8aeaad4f1f 100644 --- a/docs/reference/ilm/set-up-lifecycle-policy.asciidoc +++ b/docs/reference/ilm/set-up-lifecycle-policy.asciidoc @@ -1,7 +1,7 @@ [role="xpack"] [testenv="basic"] [[set-up-lifecycle-policy]] -== Set up {ilm} policy +== Create lifecycle policy In order for an index to use an {ilm} policy to manage its lifecycle we must first define a lifecycle policy for it to use. The following request creates a diff --git a/docs/reference/ilm/using-policies-rollover.asciidoc b/docs/reference/ilm/using-policies-rollover.asciidoc index 1670c8eaf1ac9..a9e42fbee5b86 100644 --- a/docs/reference/ilm/using-policies-rollover.asciidoc +++ b/docs/reference/ilm/using-policies-rollover.asciidoc @@ -1,7 +1,7 @@ [role="xpack"] [testenv="basic"] [[using-policies-rollover]] -== Using policies to manage index rollover +== Roll over automatically The rollover action enables you to automatically roll over to a new index based on the index size, document count, or age. When a rollover is triggered, a new