[Meta] Data Lifecycle Management

[andreidan]

Description

The Data Lifecycle Management (DLM) service handles the lifecycle of data streams based on a declarative configuration that describes the desired end state.

Phase 1:

Before removing the feature flag

• Introduce the Lifecycle model in existing data stream configuration (including the feature flag) @gmarouli (Introduce Lifecycle model in existing data stream configuration #93652)
• Introduce the cluster.dlm.default.rollover cluster setting and enable the data stream GET to display the configuration @gmarouli ([DLM] Introduce default rollover cluster setting & expose it via APIs #94240)
• PUT/GET/DELETE _data_stream/logs/_lifecycle APIs @gmarouli ( Add new endpoints to configure data lifecycle on a data stream level. #94590 )
• Create a manage_dlm privilege and use it for the _lifecycle APIs @masseyke ( Adding manage_dlm privilege #95512 )
• Introduce the DLM module @andreidan
• Implement the DataLifecycleManagementService @andreidan (Initial implementation for DataLifecycleService #94012, Add an erorr store to record the errors encountered by DLM #94391)
• Implement the DLM Service security model
• DataLifecycleManagementService execution error tracking @andreidan (Add an erorr store to record the errors encountered by DLM #94391)
• The POST indexname/_lifecycle/explain API (note it should report the errors from execution too) @andreidan ( Introduce a _lifecycle/explain API for data stream backing indices #94621 )
• Optionally add the index.lifecycle.prefer_ilm setting (defaults to true) @andreidan ( Introduce the index.lifecycle.prefer_ilm setting #95423 )
• Introduce the include_defaults parameter to the simulate index template and simulate index APIs @gmarouli ([DLM] Extend the simulate template API to support include defaults #94861)
• Add origination_date support (similar to ILM) Adding origination date to DLM #95113 @masseyke
• Implement support for automatic rollover parameter for max_age @gmarouli (Dlm add auto rollover condition max age #94950)
• Implement executing forcemerge for a backing index on the first run after rollover @masseyke (Add auto force merge functionality to DLM #95204)
• Document the feature in a holistic way (linking to APIs and such) @gmarouli (Add DLM related attributes docs#2658, Start with data stream lifecycle documentation #95326, ...)

Phase 2:

• Compose lifecycle from component templates @gmarouli (Composing data lifecycle in index templates #96027)
• Allow lifecycle and data retention to be explicitely nullyfied @gmarouli (Allow for the data lifecycle to be explicitly nullified #95979)
• Add telemetry for DLM @masseyke (Adding data_lifecycle to the _xpack/usage API #96177)
• Rebranding DLM to Data Stream Lifecycle #96875

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[andreidan]

This work is complete 🎉