kvserver: replicas do not acquire a lease and quiesce after restart

[kvoli]

Describe the problem

After restarting, the majority of replicas do not acquire a lease and remain ticking rather than quiescent. This is despite no foreground traffic being present that would tick the ranges.

The number of leaseholders still steadily increases, however at a rate of 0.02 per second, which in a cluster with a large number of ranges will take multiple days.

To Reproduce

# Note that this partially reproduces the issue, where it stalls between 1-20k. 
roachprod create $email -n 1 --gce-machine-type=n1-standard-4
roachprod stage $email release v22.1.1
roachprod start $email
roachprod run $email:1 -- 'for x in {1..48000}; do echo "create table t$x(id uuid primary key);" >> tables; done'
roachprod run $email:1 -- './cockroach sql --file tables --insecure'
sleep 600
roachprod stop $email
sleep 600
roachprod start $email

Expected behavior
Ranges should quiesce after a restart, meaning they acquire their lease.

Environment:

• CockroachDB version v22.1.1
• Server OS: Ubuntu v20.04.4 LTS

Additional context

50-70% Idle CPU usage, mostly spent in ticking ranges.

Jira issue: CRDB-17073

Jira issue: CRDB-19917

[blathers-crl]

cc @cockroachdb/replication

[kvoli]

This repro is not exactly what we've seen in the wild.

It does stall, however it reaches a higher number of leases before grinding to a glacial pace:

It will still take multiple days for the replicas to all acquire a lease in this repro, we stall with 13k remaining.

The events where we have seen this occur with a much lower % of leases acquired and then stall were in (1) ranges with lots of data (not the case here) and (2) cpu constrained environments (1 vcpu for 20k tables).

[aayushshah15]

We discussed this in the KV weekly meeting and decided that we should first understand how it is that a fraction of the ranges are acquiring leases but the other fraction aren't.

My intuition here is that its one of the queues deciding to process some replicas but not the others. In particular, I think its something like the consistency checker queue that only processes ranges once every 24 hours, so it won't reprocess the fraction of replicas that have already been checked in the last 24 hours.

@kvoli one way to validate or invalidate that theory would be to disable the consistency checker queue in your reproduction and see if none or very few of the ranges end up getting leases. If so, we'll know this is all copacetic and maybe we can consider one of the approaches we discussed in the team meeting.

[kvoli]

Further Investigations

Running some more testing. With the consistency queue enabled and disabled, we see that the initial rise of leaseholders and quiescence is not in large part contributed to by the consistency queue. However after 70 minutes this ceases, and the cluster with the consistency queue enabled continues to add leaseholders at a slow pace whilst the cluster without adds 0.

yellow = rate(raft log queue successes)
red = rate(leaseholders)
blue = rate(consistency queue successes)

With Consistency Queue

Without Consistency Queue

The initial increase, before stalling is correlated to the raft log queue processing successes. The cluster with the consistency queue enabled has a slightly higher rate of leases added, which can be attributed to the queue processing.

When looking at the logs, the last SQL stats job completed at 14:39, which corresponds to when the rate of lease acquisition went to 0 for the cluster with no consistency queue.

I220629 14:39:58.816215 5628854 jobs/registry.go:1134 ⋮ [n1] 85890  AUTO CREATE STATS job 774777073713840129: stepping through state succeeded with error: <nil>

It seems apparent then that only the SQL stats job and consistency queue are forcing lease acquisition and allowing replicas to quiesce, when there is no foreground activity on those ranges.

Problem

So from the above, it appears that cockroach has no mechanism specifically to quiesce replicas following a restart. Rather, we rely on foreground traffic, sql stats collection and queue processing to force lease acquisition, which enables quiescing when there's no activity following.

We should instrument a method for ranges to quiesce after a restart that is independent and targets this specific problem.

@aayushshah15 had two possible solutions, at a high level: forcing the lease acquisition on the raft leader, to enable quiescing or allow quiescing without needing a leaseholder for the range.

[petermattis]

I recall in the distant past that the replicate queue acquired the lease (if a range didn't hold the lease) in order to determine whether rebalancing/up-replication was required. Does that no longer take place?

[kvoli]

I recall in the distant past that the replicate queue acquired the lease (if a range didn't hold the lease) in order to determine whether rebalancing/up-replication was required.

Replicas without leases will currently not be added to the replicate_queue by the replicaScanner. Only manually enqueuing will add them. So I don't think these problem replicas are even being added to begin with.

It will when processing, however it requires them to have a computed action for that range (store imbalance, decom, upreplication etc)

Does that no longer take place?

In the process loop it still does try and acquire a lease.

refs

Acquire a lease if possible when processing a replica:

cockroach/pkg/kv/kvserver/replicate_queue.go

Line 532 in c13a702

if _, pErr := repl.redirectOnOrAcquireLease(ctx); pErr != nil {

Manually enqueuing seems to also try and acquire if possible:

cockroach/pkg/kv/kvserver/store.go

Line 3376 in c13a702

hasLease, pErr := repl.getLeaseForGossip(ctx)