Refactor SnapshotsInProgress to Use RepositoryId for Concurency Logic

[original-brownbear]

This refactors the snapshots-in-progress logic to work from RepositoryShardId when working out what parts of the repository are in-use by writes for snapshot concurrency safety. This change does not go all the way yet on this topic and there are a number of possible follow-up further improvements to simplify the logic that I'd work through over time.
But for now this allows fixing the remaining known issues that snapshot stress testing surfaced when combined with the fix in #75530.

These issues all come from the fact that ShardId is not a stable key across multiple snapshots if snapshots are partial. The scenarios that are broken are all roughly this:

• snapshot-1 for index-A with uuid-A runs and is partial
• index-A is deleted and re-created and now has uuid-B
• snapshot-2 for index-A is started and we now have it queued up behind snapshot-1 for the index
• snapshot-1 finishes and the logic tries to start the next snapshot for the same shard-id
  • this fails because the shard-id is not the same, we can't compare index uuids, just index name + shard id
  • this change fixes all these spots by always taking the round trip via RepositoryShardId

planned follow-ups here are:

• dry up logic across cloning and snapshotting more as both now essentially run the same code in many state-machine steps
• serialize snapshots-in-progress efficiently instead of re-computing the index and by-repository-shard-id lookups in the constructor every time
  • refactor the logic in snapshots-in-progress away from maps keyed by shard-id in almost all spots to this end, just keep an index name to Index map to work out what exactly is being snapshotted
• refactoring snapshots-in-progress to be a map of list of operations keyed by repository shard id instead of a list of maps as it currently is to make the concurrency simpler and more obviously correct

closes #75423

relates (#75339 ... should also fix this, but I have to verify by testing with a backport to 7.x)

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[DaveCTurner]

I rebased the stress tests on top of this branch, see 3ea18f80f99ec5f420e8f91f921d9eb0a862c850, and I still got the same Missing assignment assertion to trip 😢

testoutput-1626863771.tar.gz

[original-brownbear]

@DaveCTurner that one should be fixed by #75530 I believe. The index lookup there is broken and incorrectly interprets a re-created index with changed UUID as a deleted index still existing.

Still seeing a different exception locally though about unknown completion listeners ... looking into that now as well.

[DaveCTurner]

Hmm I merged #75530 into my branch, see 8fe45fa, and still the Missing assignment assertion trips:

testoutput-1626866221.tar.gz

[original-brownbear]

The above was caused by 064f45a.

Also I unfortunately found #75598 as a result of going through this

[original-brownbear]

@DaveCTurner to me it looks like this branch fixes the stress test now with the recent changes to master merged in. I couldn't reproduce any failures that didn't seem to be issues with the test (trying to clone indices that weren't successfully snapshotted).
Maybe take a look when you have a chance :)? This PR would be the minimal changset that I can see that would deal with the various issues around index uuids changing across snapshots but there's more cleanup planned to this code. We really should get away from the list of maps data structure we're currently using to make it obvious what operation touches what shard at any point without doing a bunch of looping to figure it out.

[DaveCTurner]

Sure, this LGTM (and to a fixed version of the stress tests too it seems)

[original-brownbear]

Thanks David!