roachtest: lease-preferences/full-first-preference-down failed

[cockroach-teamcity]

roachtest.lease-preferences/full-first-preference-down failed with artifacts on release-23.1.9-rc @ 2ce2de1b13b43c5371d4458ec36106a672694461:

(test_runner.go:1073).runTest: test timed out (1h0m0s)
test artifacts and logs in: /artifacts/lease-preferences/full-first-preference-down/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-30766

[kvoli]

We added liveness lease pinning in #108517, which was meant to make this test fail only on regressions. I'm suspecting that wasn't the original cause of #108425 (described in #108512).

I'll take another look.

[kvoli]

The node which needs to transfer leases away is n5. The logs indicate it queueing the replicas into the replicate queue, however we don't have kv-distribution logs for the period where this occurs (too many logs overwriting previous logs?). It seems entirely likely that n5 attempted to transfer the leases.

I230819 07:35:57.804490 197 kv/kvserver/replica_proposal.go:276 ⋮ [T1,n5,s5,r292/1:‹/Table/106/1/5{62063…-80491…}›,raft] 16146  new range lease repl=(n5,s5):1 seq=16 start=1692430557.803023491,0 epo=1 pro=1692430557.803023491,0 following repl=(n2,s2):8 seq=15 start=1692430498.481098297,0 epo=1 pro=1692430498.482736053,0
I230819 07:35:57.804621 197 kv/kvserver/replica_proposal.go:478 ⋮ [T1,n5,s5,r292/1:‹/Table/106/1/5{62063…-80491…}›,raft] 16147  acquired lease violates lease preferences, enqueuing for transfer [lease=repl=(n5,s5):1 seq=16 start=1692430557.803023491,0 epo=1 pro=1692430557.803023491,0 preferences=[{[‹+dc=1›]} {[‹+dc=2›]}]]

#108517 didn't work here. The timeseries shows the liveness lease on n1, so it could be the same problem as #108512.

[kvoli]

#108517 wasn't backported to release-23.1.9-rc. That explains the failure. Opened the backport here #109129.

[kvoli]

Closed on #109129