kvserver: single node unquiesces on every tick

[tbg]

Describe the problem

On a single-node cluster, user ranges with moderate traffic (~10qps) emit ~5 empty raft entries per second because this code

cockroach/pkg/kv/kvserver/replica_raft_quiesce.go

Lines 80 to 82 in fa32402

	if log.V(3) {
	log.Infof(ctx, "unquiescing %d: waking leader", r.RangeID)
	}

fires erroneously on every raft tick (200ms). This could affect single-node peak performance.

To Reproduce

$ ./cockroach start-single-node --logtostderr --insecure --vmodule=replica_application_state_machine=1 2>&1 | grep -F 'no-op on empty Raft entry'
[...]
I210408 08:33:53.850172 278 kv/kvserver/replica_application_state_machine.go:446 ⋮ [n1,s1,r37/1:‹/{Table/53-Max}›] 173  applying command with forced error: ‹kv/kvserver/replica_application_state_machine.go›:157: no-op on empty Raft entry
I210408 08:33:54.050195 303 kv/kvserver/replica_application_state_machine.go:446 ⋮ [n1,s1,r37/1:‹/{Table/53-Max}›] 174  applying command with forced error: ‹kv/kvserver/replica_application_state_machine.go›:157: no-op on empty Raft entry
I210408 08:33:54.250296 307 kv/kvserver/replica_application_state_machine.go:446 ⋮ [n1,s1,r37/1:‹/{Table/53-Max}›] 175  applying command with forced error: ‹kv/kvserver/replica_application_state_machine.go›:157: no-op on empty Raft entry
I210408 08:33:54.450539 232 kv/kvserver/replica_application_state_machine.go:446 ⋮ [n1,s1,r37/1:‹/{Table/53-Max}›] 176  applying command with forced error: ‹kv/kvserver/replica_application_state_machine.go›:157: no-op on empty Raft entry
I210408 08:33:54.650708 228 kv/kvserver/replica_application_state_machine.go:446 ⋮ [n1,s1,r37/1:‹/{Table/53-Max}›] 177  applying command with forced error: ‹kv/kvserver/replica_application_state_machine.go›:157: no-op on empty Raft entry
I210408 08:33:54.850304 322 kv/kvserver/replica_application_state_machine.go:446 ⋮ [n1,s1,r37/1:‹/{Table/53-Max}›] 178  applying command with forced error: ‹kv/kvserver/replica_application_state_machine.go›:157: no-op on empty Raft entry
I210408 08:33:54.906795 249 kv/kvserver/replica_application_state_machine.go:446 ⋮ [n1,s1,r35/1:‹/Table/{39-40}›] 179  applying command with forced error: ‹kv/kvserver/replica_application_state_machine.go›:157: no-op on empty Raft entry
[...]

Over a 60s interval, I'm counting (only for r37, the one getting the load) 291 instances, i.e. ~5 per second.

./cockroach debug raft-log cockroach-data/ 37 shows these empty entries and also tells me that the term remains stable, i.e. these entries are not emitted as a result of raft spuriously re-electing itself as a leader.

I then went and looked for places where we propose empty commands, and voila:

cockroach/pkg/kv/kvserver/replica_raft_quiesce.go

Lines 80 to 82 in fa32402

	if log.V(3) {
	log.Infof(ctx, "unquiescing %d: waking leader", r.RangeID)
	}

I verified that this fires just before each empty command, i.e. it's conclusively the origin.

Environment:

• master branch at 2c50690

Additional context
Stumbled upon this in #62791

Jira issue: CRDB-6512

Epic CRDB-25209

[erikgrinaker]

In some sense, this is expected. We aggressively quiesce ranges as soon as we can, and the only real pacing is replication and application lag, which is ~0 with a single replica. We also see this on regular RF=3 clusters, where ranges are constantly quiescing and unquiescing under moderate load.

We could add another pacing mechanism here, e.g. only quiesce if we've gone 5 ticks without any proposals. Is something like that what you had in mind here?

Alternatively, we could skip the proposal when we believe ourselves to be the leader and don't know about any other replicas, since there's noone else to wake up.

[erikgrinaker]

Submitted #103266 to only quiesce after 4 ticks (2 seconds) with no proposals. I've seen this have non-negligible cost with steady write load, both in single- and multi-node clusters.