Make snapshot deletion faster

[piyushdaftary]

Elasticsearch version (bin/elasticsearch --version): 7.6 onwards

JVM version (java -version): Java 14

OS version (uname -a if on a Unix-like system): CentOs

In Elasticsearch, snapshot deletion is a multithreaded synchronous master node operation. The sequence of the delete operation goes as follows:

1. Master node receives snapshot deletion request and registers a listener for snapshot deletion.
2. If snapshot (for which delete request is received) is in progress, stop the shard snapshot 
3. Update the cluster state with snapshot deletion entry (SnapshotDeletionsInProgress).
4. Fetch snapshot entry from s3 repository.
5. Get list of all children folder in repository under indices folder path
6. Update the shard state metadata in repository for all shards of the snapshot that is to be deleted and compute shards to be deleted from repository
7. Remove the snapshot from the list of existing snapshots in repository
8. Update new generations: 
    1. Update the index shard generations (new  generations) of all updated shard folders to next generation in Repository 
    2. Update the index-N to this new  generations in repository and make index.latest point to this new Index-N
    3. Update the new  generations in cluster state 
9. Run Cleanup operation 
    1. Delete repository root level snap-UUID and meta-UUID file of deleting snapshot 
    2. Delete stale indices folder from repository which are not referenced by any snapshots
    3. List shard level list of files to be deleted from repository
    4. Remove shard level files (in bulk delete fashion) from repository that are not referenced 
10. Update the cluster state by removing the snapshot entry and call back on the listeners
11. Listener called back which in turn responds back to the user indicating snapshot deletion is complete

Current implementation of "step 9.2 : Delete stale indices" : cleanupStaleIndices() is very slow . Snapshot deletion code deletes each stale indices from repository one after another synchronously .

With current implementation we tried to measure time taken to delete a snapshot for a cluster with 3 masters of type r5.12xlarge and 50 data nodes of type i3.4xlarge with 1601 indices and 8001 shards and 4.8 TB data. IT takes approximately 31 minutes to delete such snapshot.

	Shards #	Indices #	Snapshot Creation Time(Avg)	Snapshot Deletion Time(Avg)
Current Implementation	8001	1601	8.4 min	31.1 min

Current flow diagram of cleanupStaleIndices :

This step to cleanup stale indices can be speed up with either of the following approaches :

Suggested Optimizations

Approach1 :

Instead of making snapshot delete of stale indices a single threaded operation we make it multithreaded operation and delete multiple stale indices in parallel using SNAPSHOT thread pool's workers.
When deletion of all the stale indices are complete , we return back the DeleteResult as response of method cleanupStaleIndices()

With above Approach1 time taken to delete a snapshot for similar cluster with 3 masters of type r5.12xlarge and 50 data nodes of type i3.4xlarge with 1601 indices and 8001 shards and 4.8 TB data. IT takes approximately only 9.8 minutes to delete such snapshot.

I noted system resource utilizations such is CPU, system memory of master node. There was no major change in resource utilizations in Approach1 compared to Current implementation

Approach1 Optimization Vs Current Implementation comparison :

	Shards #	Indices #	Snapshot Creation Time(Avg)	Snapshot Deletion Time(Avg)
Approach1 Optimization	8001	1601	6.9 min	9.86 min
Current Implementation	8001	1601	6.8 min	31.1 min

Approach1 Optimization flow diagram of cleanupStaleIndices :

Approach2 :

Instead of deleting snapshot stale indices synchronously, we can completely make the method cleanupStaleIndices() asynchronous . When method is invoked to delete list of stale indices, send back the response immediately and do the snapshot deletion of stale indices in background (Using SNAPSHOT threadpool workers ).

With above Approach2 time taken to delete a snapshot for similar cluster with 3 masters of type r5.12xlarge and 50 data nodes of type i3.4xlarge with 1601 indices and 8001 shards and 4.8 TB data. IT takes approximately only 8 seconds to delete such snapshot.

	Shards #	Indices #	Snapshot Creation Time(Avg)	Snapshot Deletion Time(Avg)
Approach2 Optimization	8001	1601	6.8 min	8 sec
Current Delete	8001	1601	8.4 min	31.1 min

In Approach2 if because of master node failure, if deletion of stale indices fails , then these stale indices will be deleted in next snapshot deletion iteration , as these are stale indices and are not been referenced by any snapshots.

To track the progress of stale indices cleanup in background, a new status can be added in cluster state (Open for suggestion on how we can track the progress of stale indices cleanup)

**Approach2 Optimization flow diagram of cleanupStaleIndices : **

I want to take feedback from community on above 2 snapshot deletion optimization approaches before raising PR .

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Snapshot/Restore)

[original-brownbear]

Thanks for raising this @piyushdaftary The fact that we only execute this single-threaded really isn't all that optimal I agree and can lead to some very suboptimal situations as in your example.
I would also agree that parallelising this across indices on the SNAPSHOT pool is a reasonable fix. We should go with option 1 here and not push the execution in the background (I in fact suggested doing that in the past and we rejected the idea eventually for various user experience reasons). If you want, feel free to give option 1 a go :) If not I'm happy to implement this quickly as well next week, it should be a fairly simple change I think.

[piyushdaftary]

Thanks @original-brownbear . I will raise the PR with implementation of Approach1.

[ls-ivan-kiselev]

Ugh, I suffer from this slowness so much right now, thanks for raising it!

I have a snapshot repo to clean up of 3 years of snapshots every 2 hours and well so far it goes with a speed of 2 snapshots a day.