Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Promote the Data stream lifecycle feature to Technical Preview #100187

Merged
merged 7 commits into from
Oct 3, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions docs/changelog/100187.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
pr: 100187
summary: GA the data stream lifecycle
area: Data streams
type: "feature"
issues: []
highlight:
title: The data stream lifecycle is now in Technical Preview
body: "This marks the data stream lifecycle as available in Technical Preview.
Data streams will be able to take advantage of a built-in simplified and
resilient lifecycle implementation. Data streams with a configured lifecycle will
be automatically rolled over and tail merged (a forcemerge implementation that's
lightweight and only merges the long tail of small segments instead of the
whole shard). With the shard and index maintenance tasks being handled automatically
to ensure optimum performance, and trade-off between indexing and searching,
you'll be able to focus on the business related lifecycle aspects like data
retention."
notable: true
2 changes: 2 additions & 0 deletions docs/reference/data-management.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,8 @@ To help you manage your data, {es} offers you:
* <<data-stream-lifecycle, Data stream lifecycle>> which is the built-in lifecycle of data streams and addresses the most
common lifecycle management needs.

preview::["The built-in data stream lifecycle is in technical preview and may be changed or removed in a future release. Elastic will apply best effort to fix any issues, but this feature is not subject to the support SLA of official GA features."]

**{ilm-init}** can be used to manage both indices and data streams and it allows you to:

* Define the retention period of your data. The retention period is the minimum time your data will be stored in {es}.
Expand Down
4 changes: 4 additions & 0 deletions docs/reference/data-streams/data-stream-apis.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,13 @@ The following APIs are available for managing <<data-streams,data streams>>:
The following APIs are available for managing the built-in lifecycle of data streams:

* <<data-streams-put-lifecycle,Update data stream lifecycle>>
preview:[]
* <<data-streams-get-lifecycle,Get data stream lifecycle>>
preview:[]
* <<data-streams-delete-lifecycle,Delete data stream lifecycle>>
preview:[]
* <<data-streams-explain-lifecycle,Explain data stream lifecycle>>
preview:[]

The following API is available for <<tsds,time series data streams>>:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@
<titleabbrev>Delete Data Stream Lifecycle</titleabbrev>
++++

preview:[]

Deletes the lifecycle from a set of data streams.

[[delete-lifecycle-api-prereqs]]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@
<titleabbrev>Explain Data Stream Lifecycle</titleabbrev>
++++

preview:[]

Retrieves the current data stream lifecycle status for one or more data stream backing indices.

[[explain-lifecycle-api-prereqs]]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@
<titleabbrev>Get Data Stream Lifecycle</titleabbrev>
++++

preview:[]

Gets the lifecycle of a set of data streams.

[[get-lifecycle-api-prereqs]]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@
<titleabbrev>Put Data Stream Lifecycle</titleabbrev>
++++

preview:[]

Configures the data stream lifecycle for the targeted data streams.

[[put-lifecycle-api-prereqs]]
Expand Down
2 changes: 2 additions & 0 deletions docs/reference/data-streams/lifecycle/index.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@
[[data-stream-lifecycle]]
== Data stream lifecycle

preview:[]

A data stream lifecycle is the built-in mechanism data streams use to manage their lifecycle. It enables you to easily
automate the management of your data streams according to your retention requirements. For example, you could configure
the lifecycle to:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@
[[tutorial-manage-existing-data-stream]]
=== Tutorial: Update existing data stream

preview:[]

To update the lifecycle of an existing data stream you do the following actions:

. <<set-lifecycle>>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@
[[tutorial-manage-new-data-stream]]
=== Tutorial: Create a data stream with a lifecycle

preview:[]

To create a data stream with a built-in lifecycle, follow these steps:

. <<create-index-template-with-lifecycle>>
Expand Down Expand Up @@ -146,4 +148,4 @@ DELETE _index_template/my-index-template
--------------------------------------------------
// TEST[continued]

//////////////////////////
//////////////////////////
23 changes: 22 additions & 1 deletion docs/reference/indices/get-data-stream.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,16 @@ Name of the backing index. For naming conventions, see
`index_uuid`::
(string)
Universally unique identifier (UUID) for the index.

`prefer_ilm`::
(boolean)
Functionality in preview:[]. Indicates if this index is configured to prefer {ilm}
when both {ilm-cap} and <<data-stream-lifecycle, Data stream lifecycle>> are configured to
manage this index.

`managed_by`::
(string)
Functionality in preview:[]. Indicates the system that managed this index.
=====

`generation`::
Expand Down Expand Up @@ -206,6 +216,17 @@ NOTE: A data stream's backing indices may be assigned different lifecycle
policies. To retrieve the lifecycle policy for individual backing indices,
use the <<indices-get-settings,get index settings API>>.

`next_generation_managed_by`::
(string)
Functionality in preview:[]. Indicates the system that will managed the next generation index
(i.e. the next data stream write index).

`prefer_ilm`::
(boolean)
Functionality in preview:[]. Indicates if the index template used to create the data
stream's backing indices is configured to prefer {ilm-cap} when both {ilm-cap} and
<<data-stream-lifecycle, Data stream lifecycle>> are configured to manage this index.

`hidden`::
(Boolean) If `true`, the data stream is <<multi-hidden,hidden>>.

Expand All @@ -225,7 +246,7 @@ cluster can not write into this data stream or change its mappings.

`lifecycle`::
(object)
Contains the configuration for the data stream lifecycle management of this data stream.
Functionality in preview:[]. Contains the configuration for the data stream lifecycle management of this data stream.
+
.Properties of `lifecycle`
[%collapsible%open]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
"url":"https://www.elastic.co/guide/en/elasticsearch/reference/master/data-streams-delete-lifecycle.html",
"description":"Deletes the data stream lifecycle of the selected data streams."
},
"stability":"stable",
"stability":"experimental",
"visibility":"public",
"headers":{
"accept": [ "application/json"]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
"url": "https://www.elastic.co/guide/en/elasticsearch/reference/current/data-streams-explain-lifecycle.html",
"description": "Retrieves information about the index's current data stream lifecycle, such as any potential encountered error, time since creation etc."
},
"stability": "stable",
"stability": "experimental",
"visibility": "public",
"headers": {
"accept": [
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
"url":"https://www.elastic.co/guide/en/elasticsearch/reference/master/data-streams-get-lifecycle.html",
"description":"Returns the data stream lifecycle of the selected data streams."
},
"stability":"stable",
"stability":"experimental",
"visibility":"public",
"headers":{
"accept": [ "application/json"]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
"url":"https://www.elastic.co/guide/en/elasticsearch/reference/master/data-streams-put-lifecycle.html",
"description":"Updates the data stream lifecycle of the selected data streams."
},
"stability":"stable",
"stability":"experimental",
"visibility":"public",
"headers":{
"accept": [ "application/json"]
Expand Down