[DOCS] Document how to switch ILM policies

[jrodewig]

To switch an index's lifecycle policy, you must first remove the existing policy. Otherwise, phase execution for the index may silently fail.

Closes #70151

Preview

• Switch ILM policy: https://elasticsearch_73967.docs-preview.app.elstc.co/guide/en/elasticsearch/reference/master/set-up-lifecycle-policy.html#switch-lifecycle-policies
• Change dynamic index setting for a data stream: https://elasticsearch_73967.docs-preview.app.elstc.co/guide/en/elasticsearch/reference/master/data-streams-change-mappings-and-settings.html#change-dynamic-index-setting-for-a-data-stream

[elasticmachine]

Pinging @elastic/es-core-features (Team:Core/Features)

[elasticmachine]

Pinging @elastic/es-docs (Team:Docs)

[andreidan]

Thanks James, this LGTM.

Would it make sense to add a big warning that the remove policy API removes the policy (and all the ILM metadata) immediately without any consideration w.r.t where the index is in its lifecycle (mid-action executing for eg.) - it's the equivalent of kill -9. As an example - it could happen that for an index that is in the forcemerge action (a forcemerge action that changes the compression codec closes and reopens the index) after removing the policy using the API the index is closed.
So maybe besides a warning we should advise a manual/visual review of the index state before assigning a new policy?

[jrodewig]

Thanks @andreidan. I added a middle step to cover your suggestion.

[jrodewig]

@elasticmachine run elasticsearch-ci/docs