kv: bypass lease transfer safety checks during joint consensus

[nvanbenschoten]

This commit adds logic to bypass lease transfer safety checks (added in 034611b) when in a joint configuration and transferring the lease from a VOTER_DEMOTING to a VOTER_INCOMING. We do so because we could get stuck without a path to exit the joint configuration if we rejected this lease transfer while waiting to confirm that the target is up-to-date on its log. That confirmation may never arrive if the target is dead or partitioned away, and while we'd rather not transfer the lease to a dead node, at least we have a mechanism to recovery from that state. We also just sent the VOTER_INCOMING a snapshot (as a LEARNER, before promotion), so it is unlikely that the replica is actually dead or behind on its log.

A better alternative here would be to introduce a mechanism to choose an alternate lease transfer target after some amount of time, if the lease transfer to the VOTER_INCOMING cannot be confirmed to be safe. We may do this in the future, but given the proximity to the release and given that this matches the behavior in v22.1, we choose this approach for now.

Release note: None

Release justification: Needed to resolve release blocker.

Fixes: #88667
See also #89564

[cockroach-teamcity]

This change is 

[aayushshah15]

Do you think we should add a unit test that asserts that the lease transfer to a dead VOTER_INCOMING works?

[nvanbenschoten]

Yes, an end-to-end unit test was needed here. We were already testing that the new flag was properly set in a few tests, but not that this allowed the range to exit a joint configuration when the VOTER_INCOMING is dead.

I've added a test in a new commit. It's flaky now without #89221 because the VOTER_DEMOTING is allowed to re-acquire the lease (the starvation we hypothesized), which @shralex is working on. We'll need to sequence these changes appropriately.

[nvanbenschoten]

TFTR! I'll have to wait for #89564 to merge before this will pass CI.

[nvanbenschoten]

bors r=shralex

[craig]

Build succeeded:

• Bazel Essential CI (Cockroach)

[blathers-crl]

Encountered an error creating backports. Some common things that can go wrong:

1. The backport branch might have already existed.
2. There was a merge conflict.
3. The backport branch contained merge commits.

You might need to create your backport manually using the backport tool.

---

error creating merge commit from 0036160 to blathers/backport-release-22.2-89340: POST https://api.github.com/repos/cockroachdb/cockroach/merges: 409 Merge conflict []

you may need to manually resolve merge conflicts with the backport tool.

Backport to branch 22.2.x failed. See errors above.

---

error creating merge commit from 0036160 to blathers/backport-release-22.2.0-89340: POST https://api.github.com/repos/cockroachdb/cockroach/merges: 409 Merge conflict []

you may need to manually resolve merge conflicts with the backport tool.

Backport to branch 22.2.0 failed. See errors above.

---

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan.}

[shermanCRL]

Pardon if this was already discussed, should this backport to v22.1?

[nvanbenschoten]

These lease transfer safety checks were only added in v22.2 (in 034611b), so this doesn't need a backport. We were hopeful in #82758 that the change would be non-disruptive and eventually qualify for a backport, but as with many changes in this area, the effects are far-reaching, so we did not gain enough confidence to meet the backport criteria.