roachtest: failover/system-non-liveness/blackhole-recv/lease=expiration failed

[cockroach-teamcity]

roachtest.failover/system-non-liveness/blackhole-recv/lease=expiration failed with artifacts on master @ 795dd78a4c02f04184192b61c30bcf8fd980b5ea:

test failover/system-non-liveness/blackhole-recv/lease=expiration failed during post test assertions (see test-post-assertions.log) due to (test_runner.go:1122).func1: pq: replica unavailable: (n6,s6):5 unable to serve request to r7:/Table/{3-4} [(n6,s6):5, (n5,s5):6, (n4,s4):3, next=7, gen=18]: lost quorum (down: (n5,s5):6,(n4,s4):3); closed timestamp: 1686992400.608886409,0 (2023-06-17 09:00:00); raft status: {"id":"5","term":12,"vote":"0","commit":423,"lead":"0","raftState":"StateFollower","applied":423,"progress":{},"leadtransferee":"0"}: have been waiting 60.50s for slow proposal RequestLease [/Table/3,/Min)
test artifacts and logs in: /artifacts/failover/system-non-liveness/blackhole-recv/lease=expiration/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=2 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-28860

Epic CRDB-27234

[kvoli]

@erikgrinaker or @andrewbaptist would either of you be able to dig into this failure?

[erikgrinaker]

I'll pick it up.

[blathers-crl]

cc @cockroachdb/replication

[erikgrinaker]

Dupe of #104694.