kvserver: unquiesce proposal always errors

[erikgrinaker]

When we unquiesce a Raft group we submit an empty proposal:

cockroach/pkg/kv/kvserver/replica_raft_quiesce.go

Line 91 in 76afb00

data := raftlog.EncodeRaftCommand(raftlog.EntryEncodingStandardWithoutAC, makeIDKey(), nil)

However, this proposal always results in an error, because its ProposerLeaseSequence is always 0:

cockroach/pkg/kv/kvserver/kvserverbase/forced_error.go

Lines 193 to 197 in 736a67e

	nlhe := kvpb.NewNotLeaseHolderError(
	*replicaState.Lease, 0 /* proposerStoreID */, replicaState.Desc,
	fmt.Sprintf(
	"stale proposal: command was proposed under lease #%d but is being applied "+
	"under lease: %s", raftCmd.ProposerLeaseSequence, replicaState.Lease))

The construction of such errors has a non-negligible cost. During expiration-based lease testing with eager lease extensions, on an idle cluster with 20.000 lease extensions every 3 seconds, these accounted for 10% of CPU usage.

There is a related TODO here:

cockroach/pkg/kv/kvserver/raftlog/entry.go

Lines 188 to 190 in 4df47f5

	// TODO(tbg): can len(payload)==0 if we propose an empty command to wake up leader?
	// If so, is that a problem here?
	return errors.Wrap(protoutil.Unmarshal(raftCmdBytes, &e.Cmd), "unmarshalling RaftCommand")

We should treat these proposals as noops, similarly to what we already do here:

cockroach/pkg/kv/kvserver/kvserverbase/forced_error.go

Lines 97 to 108 in 736a67e

	if idKey == "" {
	// This is an empty Raft command (which is sent by Raft after elections
	// to trigger reproposals or during concurrent configuration changes).
	// Nothing to do here except making sure that the corresponding batch
	// (which is bogus) doesn't get executed (for it is empty and so
	// properties like key range are undefined).
	return ForcedErrResult{
	LeaseIndex: leaseIndex,
	Rejection: ProposalRejectionPermanent,
	ForcedError: noopOnEmptyRaftCommandErr,
	}
	}

This code was introduced recently in #76126, so it may be a new issue in 23.1. I'm therefore marking it as a GA blocker.

Jira issue: CRDB-26411

[blathers-crl]

cc @cockroachdb/replication