kvserver: enable raft.Config.ResumeReplicateBelowPendingSnapshot

[erikgrinaker]

Blocked on etcd-io/raft#110.

Note to self: address this TODO

cockroach/pkg/kv/kvserver/raft_snapshot_queue.go

Lines 131 to 143 in 2b91c38

	// TODO(dan): This is super brittle and non-obvious. In the common case,
	// this check avoids duplicate work, but in rare cases, we send the
	// learner snap at an index before the one raft wanted here. The raft
	// leader should be able to use logs to get the rest of the way, but it
	// doesn't try. In this case, skipping the raft snapshot would mean that
	// we have to wait for the next scanner cycle of the raft snapshot queue
	// to pick it up again. So, punt the responsibility back to raft by
	// telling it that the snapshot failed. If the learner snap ends up being
	// sufficient, this message will be ignored, but if we hit the case
	// described above, this will cause raft to keep asking for a snap and at
	// some point the snapshot lock above will be released and we'll fall
	// through to the logic below.
	repl.reportSnapshotStatus(ctx, repDesc.ReplicaID, err)

---

This allows leaders to resume replication to a follower that applies a snapshot below the leader's PendingSnapshot, as long as the snapshot connects the follower to the leader's log. Otherwise, if we delegate snapshot sending to a replica that's lagging behind the leader, it may send a snapshot below the leader's PendingSnapshot, and this would then be rejected even though it's a perfectly fine snapshot.

Resolves #106813.
Resolves #114349.
Epic: none

Release note (bug fix): Previously, Raft snapshots delegated to followers could be rejected by the leader after application if the delegate was lagging behind the leader. These snapshots are now accepted as long as they connect the follower to the leader's log.

[blathers-crl]

It looks like your PR touches production code but doesn't add or edit any test code. Did you consider adding tests to your PR?

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[cockroach-teamcity]

This change is