storage: preemptive snapshots

[petermattis]

Preemptive snapshots were disabled due to raft corruption issues. With #7619 fixed we should revisit.

The lack of preemptive snapshots makes the replica reservation system a lot less useful. We can see this in TestMultinodeCockroach which usually finishes within a few seconds, but sometimes takes 30-40s. The slow runs are caused by a declined reservation due to there being to many outstanding reservations. The default max outstanding reservations is 5 and TestMultinodeCockroach creates 6 ranges. The lack of preemptive snapshots causes replicateQueue.process to asynchronously perform the replication. Specifically, the call to Replica.ChangeReplicas returns before the reservation is fulfilled. The intention is that Replica.ChangeReplicas blocks until the preemptive snapshot has been processed remotely. Without that feedback, we quickly iterate over the replicas creating reservations until we hit defaultMaxReservations at which point processing fails and we have to wait until the next replica scanner iteration to replicate the range.

When reenabling preemptive snapshots, the sending of the snapshot in Replica.ChangeReplicas should be made synchronous. Currently we send the preemptive snapshot via the normal raft transport mechanism. But this only means the snapshot is queued up at the raft transport. The code then proceeds to issue the ChangeReplicas operation through Raft. Depending on timing, Raft might then request another snapshot. Regardless of this waste, the feedback loop for replication is being violated by sending the preemptive snapshot asynchronously.

See also #6144