storage: release preemptive snapshot after remote apply

[petermattis]

Previously we were releasing the snapshot (i.e. calling
Replica.CloseOutSnap()) when the ChangeReplicas operation
completed. Now we release the snapshot as soon as the remote node has
applied it. This is important to allow other ranges to make progress
which might be required for the current ChangeReplicas operation to
complete.

Fixes #10483
Fixes #10306
See #10409

---

This change is

[petermattis]

Perhaps we don't need #10482 as this PR also fixes #10409.

Not quite sure how to test this. Thoughts?

[tamird]

Perhaps you could simulate exactly the problem in #10409? We'll need a know to inject a raft log truncation after the first range on the second node receives its pre-emptive snapshot and before the ensuing conf change.

---

Reviewed 1 of 1 files at r1.
Review status: all files reviewed at latest revision, all discussions resolved, all commit checks successful.

---

Comments from Reviewable

[petermattis]

Perhaps you could simulate exactly the problem in #10409?

I should have been clearer that I made several attempts to test this change, specifically trying to simulate the problem in #10409. I'll give it another go tomorrow.

[petermattis]

Added a test that directly tests that the preemptive snapshot is released. The test fails on master and passes on this PR.

[bdarnell]

I think we should keep #10482; the discrepancy between the Commit index and the current membership is a concern even when it doesn't lead to snapshots getting stuck.

---

Review status: 1 of 2 files reviewed at latest revision, all discussions resolved, all commit checks successful.

---

Comments from Reviewable

[tamird]

Reviewed 1 of 1 files at r2.
Review status: all files reviewed at latest revision, all discussions resolved, all commit checks successful.

---

Comments from Reviewable

[petermattis]

I think we should keep #10482; the discrepancy between the Commit index and the current membership is a concern even when it doesn't lead to snapshots getting stuck.

I agree. I've already merged #10482.