Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automatically migrate default ILM policy to Data Stream Lifecycle Management #14128

Open
axw opened this issue Sep 20, 2024 · 4 comments
Open

Comments

@axw
Copy link
Member

axw commented Sep 20, 2024

In 8.15.0 we enable the apm-data plugin by default in Elasticsearch, which ships index templates that make use of data stream lifecycle by default. It is still possible to opt into ILM (except in Serverless projects, which do not support ILM).

There was a bug in 8.15.0 that we fixed in 8.15.1 (#13898): data streams upgraded from older versions would be "unmanaged" on upgrade. Data stream lifecycle is only automatically configured on newly data streams, so we need to keep some ILM configuration for existing data streams.

This leaves us in an indefinite undesirable state:

  • new data streams use data stream lifecycle
  • old data streams use ILM, unless manually migrated

We should automatically migrate existing data streams from ILM to data stream lifecycle management when the existing data streams use the default ILM policy. The retention durations are aligned, and the default ILM policy does not use any advanced functionality (e.g. hot/warm/cold tiering), so this should be considered a non-breaking change.

@pilsy
Copy link

pilsy commented Sep 24, 2024

It's not really an enhancement, it's more of a bug -- how do i migrate this myself? none of the tracing stuff works anymore in our instances... (we are spending 25k a year on them, and they no longer work)

@axw
Copy link
Member Author

axw commented Sep 24, 2024

@pilsy which part's a bug? There's definitely a bug in 8.15.0, for which there's a fix in 8.15.1: #13898

Is there something else affecting you too?

@pilsy
Copy link

pilsy commented Oct 15, 2024

@axw support eventually reached out and helped me fix the problem, basically when the cluster was upgraded to 8.15.0 all tracing, logging and metrics stopped displaying in the interface, even tho the data was still being ingested... and a simple upgrade to 8.15.1 did not automagically fix it.

@axw
Copy link
Member Author

axw commented Oct 15, 2024

@pilsy thanks, I will follow up on the support case and see what else we can/need to do. Sorry about the trouble this caused you.

EDIT: I see the resolution was to force a rollover to the data streams. This is related to elastic/elasticsearch#112781

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants