ILM delete phase deletes searchable snapshot before index

[DaveCTurner]

When ILM deletes a searchable snapshot index, it deletes the snapshot first, waits for that to complete, and then deletes the index. This means that there's a period of time when the index still exists but attempts to search it may fail with a org.elasticsearch.snapshots.SnapshotMissingException.

elasticsearch/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/ilm/DeleteAction.java

Lines 100 to 101 in f32c8f5

	CleanupSnapshotStep cleanupSnapshotStep = new CleanupSnapshotStep(cleanSnapshotKey, deleteStepKey, client);
	DeleteStep deleteStep = new DeleteStep(deleteStepKey, nextStepKey, client);

We should delete the index first, and only delete the snapshot once the index deletion is complete.

Relates #116379 which can prevent the snapshot deletion from completing properly, leaving the index in a broken state for a long time.

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[dakrone]

This is an unfortunate byproduct of the way that ILM works. If we were to delete the index prior to the snapshot, then ILM would be unmoored from any execution, and it wouldn't then proceed to remove the snapshot. We could try to do the delete and snapshot delete in the same step, but we still risk hanging snapshots during failure.

Originally, I remember that quite a while ago @tlrx was looking into something where deleting the snapshot also deleted the index at the same time, or maybe it was a way to delete the index with deleting the snapshot at the same time, @tlrx did anything come out of that?

[tlrx]

@dakrone attempt is #79156 which wasn't completed due to complexity and other priorities. The idea was to delete the snapshot once the mounted index is deleted, if ILM delete_snapshot was set to true.

[DaveCTurner]

I left a comment about a possibly-simpler approach.