Adding an ILM Action to be able to do snapshot an index during an ILM phase.

[rkalluri-clgx]

I would like to request an ability to snapshot an index as part of an ILM action (https://www.elastic.co/guide/en/elasticsearch/reference/current/_actions.html)

This would be useful for time based indices especially when using a cloud storage as repository like S3 or GCS to off load the cold data off the elastic cluster

This will provide ways to restore those indices when there is a need to restore specific time based indices.

[rkalluri-clgx]

I was thinking we can implement this as a generic user provided action / callback that can implement any custom action based on a phase of the lifecycle like providing notifications and such.

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/ILM+SLM)

[dakrone]

@rkalluri-clgx we recently implemented #45067 (#50454) which allows waiting in the delete phase for an SLM policy snapshot to be taken. Does this cover what you're asking for? (the index doesn't have to be deleted)

If not, can you clarify your exact use case?

[rkalluri-clgx]

I am thinking from the aspect of index archival not backup. This will be like a final read only copy of the index that is closed so that we can replay these back. I treat the SLM to be more for disaster recovery.

[zuketo]

Hi @rkalluri-clgx, can you please provide more details on your use case here? I'm trying to understand what "replay these back" means? Would you like to snapshot older indices automatically before deleting them, then have a mechanism for restoring those indices when needed? You may also be interested in freezing indices within ILM, frozen indices will allow you to create very dense nodes for long term data storage.

[cpmoore]

+1 for this feature
For my use case, there are certain indexes where I want to take the number_of_replicas down to 0 after the index is rolled over, however I want to ensure a snapshot exists before I do this (just in the event that the worker node where that index resides is lost beyond recovery)
Therefore i need the snapshot taken as soon as the index is rolled over, but before the index settings change is applied.

[dakrone]

Just wanted to point out to folks in this also, that we do have support in ILM for searchable snapshots now: #50806 and in that step, it can be configured to keep the snapshot past the point where the index would be deleted.

[dakrone]

For now we are closing this as we don't want to increase the number of single-snapshot indices for backup purposes, as that could cause cluster instability (see #66042)