roachtest: failover/chaos/read-write/lease=expiration failed

[cockroach-teamcity]

roachtest.failover/chaos/read-write/lease=expiration failed with artifacts on master @ 5d02bd9ff6b2bccecf6d43fc6cd647167b91f782:

(failover.go:1766).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2351).Run: context canceled
(cluster.go:2351).Run: context canceled
(cluster.go:2351).Run: context canceled
(cluster.go:2351).Run: context canceled
test artifacts and logs in: /artifacts/failover/chaos/read-write/lease=expiration/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=2
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=false
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• roachtest: failover/chaos/read-write/lease=expiration failed #121457 roachtest: failover/chaos/read-write/lease=expiration failed [C-bug C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-23.2]
• roachtest: failover/chaos/read-write/lease=expiration failed #119453 roachtest: failover/chaos/read-write/lease=expiration failed [C-bug C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-23.1.15-rc]

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-38233

[miraradeva]

Looks like node 9 crashed, and this is at the top of its stack trace:

panic: runtime error: index out of range [1390] with length 300

goroutine 135728 gp=0xc00dc9ac40 m=14 mp=0xc013d65008 [running]:
panic({0x60c44e0?, 0xc02c04af60?})
	GOROOT/src/runtime/panic.go:779 +0x158 fp=0xc00d696b00 sp=0xc00d696a50 pc=0x49bd98
runtime.goPanicIndex(0x56e, 0x12c)
	GOROOT/src/runtime/panic.go:114 +0x7c fp=0xc00d696b40 sp=0xc00d696b00 pc=0x49a6dc
github.com/cockroachdb/cockroach/pkg/storage/disk.(*monitorTracer).String(0xc0017962c0)
	github.com/cockroachdb/cockroach/pkg/storage/disk/monitor_tracer.go:140 +0x4d7 fp=0xc00d696f08 sp=0xc00d696b40 pc=0x2014e37
github.com/cockroachdb/cockroach/pkg/storage/disk.(*Monitor).LogTrace(...)
	github.com/cockroachdb/cockroach/pkg/storage/disk/monitor.go:275
github.com/cockroachdb/cockroach/pkg/storage.(*Pebble).makeMetricEtcEventListener.func4.1()
	github.com/cockroachdb/cockroach/pkg/storage/pebble.go:1460 +0x4f fp=0xc00d696fb0 sp=0xc00d696f08 pc=0x208006f
github.com/cockroachdb/cockroach/pkg/storage.(*Pebble).async.func1()
	github.com/cockroachdb/cockroach/pkg/storage/pebble.go:1378 +0x59 fp=0xc00d696fe0 sp=0xc00d696fb0 pc=0x207f4b9
runtime.goexit({})
	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00d696fe8 sp=0xc00d696fe0 pc=0x4d7e01
created by github.com/cockroachdb/cockroach/pkg/storage.(*Pebble).async in goroutine 135727
	github.com/cockroachdb/cockroach/pkg/storage/pebble.go:1376 +0x7c

Pinging @cockroachdb/storage to see if this looks familiar.

[cockroach-teamcity]

roachtest.failover/chaos/read-write/lease=expiration failed with artifacts on master @ 2b5baff23f9b53da26f0fc1a763ca43384b464b0:

(failover.go:1766).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2351).Run: context canceled
(cluster.go:2351).Run: context canceled
(cluster.go:2351).Run: context canceled
(cluster.go:2351).Run: context canceled
test artifacts and logs in: /artifacts/failover/chaos/read-write/lease=expiration/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=2
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=false
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• roachtest: failover/chaos/read-write/lease=expiration failed #121457 roachtest: failover/chaos/read-write/lease=expiration failed [C-bug C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-23.2]
• roachtest: failover/chaos/read-write/lease=expiration failed #119453 roachtest: failover/chaos/read-write/lease=expiration failed [C-bug C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-23.1.15-rc]

This test on roachdash | Improve this report!

[nicktrav]

According to @andrewbaptist, this is likely a dup of #123021.

[nicktrav]

cc: @RaduBerinde - as we have a patch up.

[jbowens]

Fixed by #123218