Skip to content

Commit

Permalink
Merge #107412
Browse files Browse the repository at this point in the history
107412: kvserver: fix 'observed raft log position' assertion r=irfansharif a=irfansharif

Fixes #107336.
Fixes #106123.
Fixes #107156.
Fixes #106589.

It's possible to hit this assertion under --stress --race when the proposing replica is starved enough for raft ticks that it loses leadership right when it steps proposals through raft. We're relying on undocumented API semantics in the etcd raft library whereby it mutates stepped entries with the term+index its to end up in. But that's only applicable if stepping through entries as a leader. Simply relax this assertion instead.

Release note: None

Co-authored-by: irfan sharif <[email protected]>
  • Loading branch information
craig[bot] and irfansharif committed Jul 24, 2023
2 parents d293bb3 + deb538f commit 24603c9
Showing 1 changed file with 10 additions and 0 deletions.
10 changes: 10 additions & 0 deletions pkg/kv/kvserver/replica_proposal_buf.go
Original file line number Diff line number Diff line change
Expand Up @@ -1041,6 +1041,16 @@ func maybeDeductFlowTokens(
if admitHandle.handle == nil {
continue // nothing to do
}
if ents[i].Term == 0 && ents[i].Index == 0 {
// It's possible to have lost raft leadership right before stepping
// proposals through raft. They'll get forwarded to the new raft
// leader, and for flow token purposes, there's no tracking
// necessary. The token deductions below asserts on monotonic
// observations of log positions, which this empty position would
// otherwise violate. There's integration code elsewhere that will
// free up all tracked tokens as a result of this leadership change.
return
}
log.VInfof(ctx, 1, "bound index/log terms for proposal entry: %s",
raft.DescribeEntry(ents[i], func(bytes []byte) string {
return "<omitted>"
Expand Down

0 comments on commit 24603c9

Please sign in to comment.