roachtest: tpccbench/nodes=12/cpu=16/lease=expiration failed

[cockroach-teamcity]

roachtest.tpccbench/nodes=12/cpu=16/lease=expiration failed with artifacts on master @ 61806f42e4c833b863ca9c2f62ce34918f1c8277:

test artifacts and logs in: /artifacts/tpccbench/nodes=12/cpu=16/lease=expiration/run_1
(test_runner.go:1154).func1: 4 dead node(s) detected

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-28680

[irfansharif]

F230610 21:05:53.796987 1920 kv/kvserver/raft_transport.go:911 â‹® [T1,n3] 50  queue for n9 does not exist
F230610 21:05:53.796987 1920 kv/kvserver/raft_transport.go:911 â‹® [T1,n3] 50 !goroutine 1920 [running]:
F230610 21:05:53.796987 1920 kv/kvserver/raft_transport.go:911 â‹® [T1,n3] 50 !github.com/cockroachdb/cockroach/pkg/util/allstacks.GetWithBuf({0x0?, 0xc002df40f0?, 0x47203f?})
F230610 21:05:53.796987 1920 kv/kvserver/raft_transport.go:911 â‹® [T1,n3] 50 !	github.com/cockroachdb/cockroach/pkg/util/allstacks/allstacks.go:32 +0x85
F230610 21:05:53.796987 1920 kv/kvserver/raft_transport.go:911 â‹® [T1,n3] 50 !github.com/cockroachdb/cockroach/pkg/util/allstacks.Get(...)
F230610 21:05:53.796987 1920 kv/kvserver/raft_transport.go:911 â‹® [T1,n3] 50 !	github.com/cockroachdb/cockroach/pkg/util/allstacks/allstacks.go:19
F230610 21:05:53.796987 1920 kv/kvserver/raft_transport.go:911 â‹® [T1,n3] 50 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(0xc00129cc20, {{{0xc0021d0030, 0x24}, {0x608286a, 0x1}, {0x6082868, 0x1}, {0x608286a, 0x1}}, 0x176768330438ba76, ...})
F230610 21:05:53.796987 1920 kv/kvserver/raft_transport.go:911 â‹® [T1,n3] 50 !	github.com/cockroachdb/cockroach/pkg/util/log/clog.go:263 +0x99
F230610 21:05:53.796987 1920 kv/kvserver/raft_transport.go:911 â‹® [T1,n3] 50 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x7633950, 0xc00882fad0}, 0x2, 0x4, 0x0, 0x0?, {0x604631c, 0x1c}, {0xc00732dd30, 0x1, ...})

Fixed by #104699. Part of #98703.