Merge #131716

131716: raft: use candidate's log term to determine inFortifyLease condition r=nvanbenschoten a=arulajmani

Previously, if a node received a Msg{,Pre}Vote request from a peer but it still supported a leader in StoreLiveness, the node would reject the request with the justification that it was fortified. However, if the campaigning peer is doing so with a higher log term than we're aware of, then this conclusively proves that the leadership term we're fortifying has actually ended. This then allows us to de-fortify, which ends our fortification lease, granting us the freedom to call elections and vote in the current (and future) campaigns. This patch recognizes this case and switches to this behavior.

Note that this is more than just an optimization -- one can craft scenarios where de-fortifying and voting in an election is the only way we'll ever elect a new leader. One such scenario is shown in the test case attached to this commit.

Epic: none

Release note: None

Co-authored-by: Arul Ajmani <[email protected]>

pkg/raft/raft.go

@@ -1418,12 +1418,20 @@ func (r *raft) Step(m pb.Message) error {
 		if m.Type == pb.MsgVote || m.Type == pb.MsgPreVote {
 			force := bytes.Equal(m.Context, []byte(campaignTransfer))
 			inHeartbeatLease := r.checkQuorum && r.lead != None && r.electionElapsed < r.electionTimeout
-			// NB: A fortified leader is allowed to bump its term. It'll need to
-			// re-fortify once if it gets elected at the higher term though, so the
-			// leader must take care to not regress its supported expiration. However,
-			// at the follower, we grant the fortified leader our vote at the higher
-			// term.
-			inFortifyLease := r.supportingFortifiedLeader() && r.lead != m.From
+			inFortifyLease := r.supportingFortifiedLeader() &&
+				// NB: A fortified leader is allowed to bump its term. It'll need to
+				// re-fortify once if it gets elected at the higher term though, so the
+				// leader must take care to not regress its supported expiration.
+				// However, at the follower, we grant the fortified leader our vote at
+				// the higher term.
+				r.lead != m.From &&
+				// NB: If the peer that's campaigning has an entry in its log with a
+				// higher term than what we're aware of, then this conclusively proves
+				// that a new leader was elected at a higher term. We never heard from
+				// this new leader (otherwise we'd have bumped r.Term in response).
+				// However, any fortification we're providing to a leader that has been
+				// since dethroned is pointless.
+				m.LogTerm <= r.Term
 			if !force && (inHeartbeatLease || inFortifyLease) {
 				// If a server receives a Request{,Pre}Vote message but is still
 				// supporting a fortified leader, it does not update its term or grant
@@ -1551,11 +1559,12 @@ func (r *raft) Step(m pb.Message) error {
 	case pb.MsgVote, pb.MsgPreVote:
 		// We can vote if this is a repeat of a vote we've already cast...
 		canVote := r.Vote == m.From ||
-			// ...we haven't voted and we don't think there's a leader yet in this term...
+			// ...OR we haven't voted and we don't think there's a leader yet in this
+			// term...
 			(r.Vote == None && r.lead == None) ||
-			// ...or this is a PreVote for a future term...
+			// ...OR this is a PreVote for a future term...
 			(m.Type == pb.MsgPreVote && m.Term > r.Term)
-		// ...and we believe the candidate is up to date.
+		// ...AND we believe the candidate is up to date.
 		lastID := r.raftLog.lastEntryID()
 		candLastID := entryID{term: m.LogTerm, index: m.Index}
 		if canVote && r.raftLog.isUpToDate(candLastID) {