diff --git a/docs/reference/glossary.asciidoc b/docs/reference/glossary.asciidoc index 07270228bace9..40ff0828afe2d 100644 --- a/docs/reference/glossary.asciidoc +++ b/docs/reference/glossary.asciidoc @@ -99,6 +99,30 @@ Follower indices are the target indices for <>. They exist in your local cluster and replicate <>. // end::follower-index-def[] +[[glossary-force-merge]] force merge :: +// tag::force-merge-def[] +// tag::force-merge-def-short[] +Manually trigger a merge to reduce the number of segments in each shard of an index +and free up the space used by deleted documents. +// end::force-merge-def-short[] +You should not force merge indices that are actively being written to. +Merging is normally performed automatically, but you can use force merge after +<> to reduce the shards in the old index to a single segment. +See the {ref}/indices-forcemerge.html[force merge API]. +// end::force-merge-def[] + +[[glossary-freeze]] freeze :: +// tag::freeze-def[] +// tag::freeze-def-short[] +Make an index read-only and minimize its memory footprint. +// end::freeze-def-short[] +Frozen indices can be searched without incurring the overhead of of re-opening a closed index, +but searches are throttled and might be slower. +You can freeze indices to reduce the overhead of keeping older indices searchable +before you are ready to archive or delete them. +See the {ref}/freeze-index-api.html[freeze API]. +// end::freeze-def[] + [[glossary-id]] id :: // tag::id-def[] The ID of a <> identifies a document. The @@ -136,6 +160,18 @@ See {ref}/indices-add-alias.html[Add index alias]. // end::index-alias-def[] -- +[[glossary-index-template]] index template :: ++ +-- +// tag::index-template-def[] +// tag::index-template-def-short[] +Defines settings and mappings to apply to new indexes that match a simple naming pattern, such as _logs-*_. +// end::index-template-def-short[] +An index template can also attach a lifecycle policy to the new index. +Index templates are used to automatically configure indices created during <>. +// end::index-template-def[] +-- + [[glossary-leader-index]] leader index :: // tag::leader-index-def[] Leader indices are the source indices for <>. They exist @@ -253,6 +289,19 @@ shard will never be started on the same node as its primary shard. // end::replica-shard-def[] -- +[[glossary-rollover]] rollover :: ++ +-- +// tag::rollover-def[] +// tag::rollover-def-short[] +Redirect an alias to begin writing to a new index when the existing index reaches a certain age, number of docs, or size. +// end::rollover-def-short[] +The new index is automatically configured according to any matching <>. +For example, if you're indexing log data, you might use rollover to create daily or weekly indices. +See the {ref}/indices-rollover-index.html[rollover index API]. +// end::rollover-def[] +-- + [[glossary-routing]] routing :: + -- @@ -292,7 +341,11 @@ nodes. [[glossary-shrink]] shrink :: // tag::shrink-def[] -To reduce the amount of shards in an index. +// tag::shrink-def-short[] +Reduce the number of primary shards in an index. +// end::shrink-def-short[] +You can shrink an index to reduce its overhead when the request volume drops. +For example, you might opt to shrink an index once it is no longer the write index. See the {ref}/indices-shrink-index.html[shrink index API]. // end::shrink-def[] diff --git a/docs/reference/ilm/getting-started-ilm.asciidoc b/docs/reference/ilm/getting-started-ilm.asciidoc index f7ca3f4d216fc..6ff32cdd2a50c 100644 --- a/docs/reference/ilm/getting-started-ilm.asciidoc +++ b/docs/reference/ilm/getting-started-ilm.asciidoc @@ -1,7 +1,7 @@ [role="xpack"] [testenv="basic"] [[getting-started-index-lifecycle-management]] -== Getting started with {ilm} +== Get started with {ilm} Let's jump into {ilm} ({ilm-init}) by working through a hands-on scenario. This section will leverage many new concepts unique to {ilm-init} that @@ -160,7 +160,7 @@ GET datastream-*/_ilm/explain The above request will retrieve {ilm-init} execution information for all our managed indices. - +[[36818c6d9f434d387819c30bd9addb14]] [source,console-result] -------------------------------------------------- { diff --git a/docs/reference/ilm/ilm-and-snapshots.asciidoc b/docs/reference/ilm/ilm-and-snapshots.asciidoc index c71fbe0471da2..5de30f11382b4 100644 --- a/docs/reference/ilm/ilm-and-snapshots.asciidoc +++ b/docs/reference/ilm/ilm-and-snapshots.asciidoc @@ -1,7 +1,7 @@ [role="xpack"] [testenv="basic"] [[index-lifecycle-and-snapshots]] -== Restoring snapshots of managed indices +== Restore a managed index When restoring a snapshot that contains indices managed by Index Lifecycle Management, the lifecycle will automatically continue to execute after the diff --git a/docs/reference/ilm/ilm-with-existing-indices.asciidoc b/docs/reference/ilm/ilm-with-existing-indices.asciidoc index 96c1e4589de9a..8def57c0bc346 100644 --- a/docs/reference/ilm/ilm-with-existing-indices.asciidoc +++ b/docs/reference/ilm/ilm-with-existing-indices.asciidoc @@ -1,7 +1,7 @@ [role="xpack"] [testenv="basic"] [[ilm-with-existing-indices]] -== Using {ilm-init} with existing indices +== Manage existing indices While it is recommended to use {ilm-init} to manage the index lifecycle from start to finish, it may be useful to use {ilm-init} with existing indices, diff --git a/docs/reference/ilm/index.asciidoc b/docs/reference/ilm/index.asciidoc index c26e698efda25..a398f56d1ce9a 100644 --- a/docs/reference/ilm/index.asciidoc +++ b/docs/reference/ilm/index.asciidoc @@ -1,72 +1,35 @@ [role="xpack"] [testenv="basic"] [[index-lifecycle-management]] -= Managing the index lifecycle += Manage the index lifecycle [partintro] -- - -The <> enable you to -automate how you want to manage your indices over time. Rather than simply -performing management actions on your indices on a set schedule, you can base -actions on other factors such as shard size and performance requirements. - -You control how indices are handled as they age by attaching a -lifecycle policy to the index template used to create them. You can update -the policy to modify the lifecycle of both new and existing indices. - -For time series indices, there are four stages in the index lifecycle: - -* Hot--the index is actively being updated and queried. -* Warm--the index is no longer being updated, but is still being queried. -* Cold--the index is no longer being updated and is seldom queried. The -information still needs to be searchable, but it's okay if those queries are -slower. -* Delete--the index is no longer needed and can safely be deleted. - -The lifecycle policy governs how the index transitions through these stages and -the actions that are performed on the index at each stage. The policy can -specify: - -* The maximum size or age at which you want to roll over to a new index. -* The point at which the index is no longer being updated and the number of -primary shards can be reduced. -* When to force a merge to permanently delete documents marked for deletion. -* The point at which the index can be moved to less performant hardware. -* The point at which the availability is not as critical and the number of -replicas can be reduced. -* When the index can be safely deleted. - -For example, if you are indexing metrics data from a fleet of ATMs into -Elasticsearch, you might define a policy that says: - -. When the index reaches 50GB, roll over to a new index. -. Move the old index into the warm stage, mark it read only, and shrink it down -to a single shard. -. After 7 days, move the index into the cold stage and move it to less expensive -hardware. -. Delete the index once the required 30 day retention period is reached. - -*Snapshot Lifecycle Management* - -ILM itself does allow managing indices, however, managing snapshots for a set of -indices is outside of the scope of an index-level policy. Instead, there are -separate APIs for managing snapshot lifecycles. Please see the -<> -documentation for information about configuring snapshots. - -See <>. - -[IMPORTANT] -=========================== -{ilm} does not support mixed-version cluster usage. Although it -may be possible to create such new policies against -newer-versioned nodes, there is no guarantee they will -work as intended. New policies using new actions that -do not exist in the oldest versioned node will cause errors. -=========================== +You can configure {ilm} ({ilm-init}) policies to automatically manage indices according to +your performance, resiliency, and retention requirements. +For example, you could use {ilm-init} to: + +* Create a new index each day, week, or month and archive previous ones +* Spin up a new index when an index reaches a certain size +* Delete stale indices to enforce data retention standards + +[TIP] +To automatically back up your indices and manage snapshots, +use <>. + +* <> +* <> +* <> +* <> +* <> +* <> +* <> +* <> +* <> +* <> -- +include::overview-ilm.asciidoc[] include::getting-started-ilm.asciidoc[] @@ -74,14 +37,15 @@ include::policy-definitions.asciidoc[] include::set-up-lifecycle-policy.asciidoc[] -include::using-policies-rollover.asciidoc[] - include::update-lifecycle-policy.asciidoc[] include::error-handling.asciidoc[] -include::ilm-and-snapshots.asciidoc[] - include::start-stop-ilm.asciidoc[] +include::using-policies-rollover.asciidoc[] + include::ilm-with-existing-indices.asciidoc[] + +include::ilm-and-snapshots.asciidoc[] + diff --git a/docs/reference/ilm/overview-ilm.asciidoc b/docs/reference/ilm/overview-ilm.asciidoc new file mode 100644 index 0000000000000..af3797587fb69 --- /dev/null +++ b/docs/reference/ilm/overview-ilm.asciidoc @@ -0,0 +1,69 @@ +[role="xpack"] +[testenv="basic"] +[[overview-index-lifecycle-management]] +== Index lifecycle management overview + +++++ +Overview +++++ + +You can create and apply {ilm-cap} ({ilm-init}) policies to automatically manage your indices +according to your performance, resiliency, and retention requirements. +Index lifecycle policies can trigger actions such as: + +* **Rollover** - +include::../glossary.asciidoc[tag=rollover-def-short] +* **Shrink** - +include::../glossary.asciidoc[tag=shrink-def-short] +* **Force merge** - +include::../glossary.asciidoc[tag=force-merge-def-short] +* **Freeze** - +include::../glossary.asciidoc[tag=freeze-def-short] +* **Delete** - Permanently remove an index, including all of its data and metadata. + +Typically, you associate a lifecycle policy with an index template so it is automatically applied +to new indices. +You can also apply a policy manually when you create an index. + +{ilm-init} simplifies managing indices in hot-warm-cold architectures, +which are common when you're working with time-series data such as logs and metrics. + +As an index ages, it moves through four possible phases: + +* Hot--the index is actively being updated and queried. +* Warm--the index is no longer being updated, but is still being queried. +* Cold--the index is no longer being updated and is seldom queried. The +information still needs to be searchable, but it's okay if those queries are +slower. +* Delete--the index is no longer needed and can safely be deleted. + +A lifecycle policy controls how an index moves between phases and +what actions to perform during each phase. You can specify: + +* The maximum size or age at which you want to roll over to a new index. +* The point at which the index is no longer being updated and the number of +primary shards can be reduced. +* When to force a merge to permanently delete documents marked for deletion. +* The point at which the index can be moved to less performant hardware. +* The point at which the availability is not as critical and the number of +replicas can be reduced. +* When the index can be safely deleted. + +For example, if you are indexing metrics data from a fleet of ATMs into +Elasticsearch, you might define a policy that says: + +. When the index reaches 50GB, roll over to a new index. +. Move the old index into the warm stage, mark it read only, and shrink it down +to a single shard. +. After 7 days, move the index into the cold stage and move it to less expensive +hardware. +. Delete the index once the required 30 day retention period is reached. + +[IMPORTANT] +=========================== +To use {ilm-init}, all nodes in a cluster must run the same version. +Although it might be possible to create and apply policies in a mixed-version cluster, +there is no guarantee they will work as intended. +Attempting to use a policy that contains actions that aren't +supported on all nodes in a cluster will cause errors. +=========================== \ No newline at end of file diff --git a/docs/reference/ilm/set-up-lifecycle-policy.asciidoc b/docs/reference/ilm/set-up-lifecycle-policy.asciidoc index e430ae9479a5a..18ed9c63d8c66 100644 --- a/docs/reference/ilm/set-up-lifecycle-policy.asciidoc +++ b/docs/reference/ilm/set-up-lifecycle-policy.asciidoc @@ -1,7 +1,7 @@ [role="xpack"] [testenv="basic"] [[set-up-lifecycle-policy]] -== Set up {ilm} policy +== Create lifecycle policy In order for an index to use an {ilm} policy to manage its lifecycle we must first define a lifecycle policy for it to use. The following request creates a diff --git a/docs/reference/ilm/using-policies-rollover.asciidoc b/docs/reference/ilm/using-policies-rollover.asciidoc index 04b084a0e8e59..ff41c29f4bc21 100644 --- a/docs/reference/ilm/using-policies-rollover.asciidoc +++ b/docs/reference/ilm/using-policies-rollover.asciidoc @@ -1,7 +1,7 @@ [role="xpack"] [testenv="basic"] [[using-policies-rollover]] -== Using policies to manage index rollover +== Roll over automatically The rollover action enables you to automatically roll over to a new index based on the index size, document count, or age. When a rollover is triggered, a new