roachtest: clearrange/checks=true/rangeTs=false failed

[cockroach-teamcity]

roachtest.clearrange/checks=true/rangeTs=false failed with artifacts on release-23.1 @ 366fe013b4b63bef98c953d03b302dc5e0b13ee7:

(cluster.go:2113).Run: context canceled
(monitor.go:153).Wait: monitor failure: unexpected node event: n7: cockroach process died (exit code 10)
test artifacts and logs in: /artifacts/clearrange/checks=true/rangeTs=false/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

Jira issue: CRDB-32408

[jbowens]

n10 fataled due to a disk stall during the first IMPORT.

F231016 12:33:16.715338 96386 storage/pebble.go:1268 ⋮ [T1,n10] 1347  file write stall detected: disk slowness detected: sync on file i81.t7 (0 bytes) has been ongoing for 20.0s
F231016 12:33:16.715338 96386 storage/pebble.go:1268 ⋮ [T1,n10] 1347 !goroutine 96386 [running]:
F231016 12:33:16.715338 96386 storage/pebble.go:1268 ⋮ [T1,n10] 1347 !github.com/cockroachdb/cockroach/pkg/util/allstacks.GetWithBuf({0x0?, 0xc002ef80f0?, 0x47203f?})
F231016 12:33:16.715338 96386 storage/pebble.go:1268 ⋮ [T1,n10] 1347 !	github.com/cockroachdb/cockroach/pkg/util/allstacks/allstacks.go:32 +0x85
F231016 12:33:16.715338 96386 storage/pebble.go:1268 ⋮ [T1,n10] 1347 !github.com/cockroachdb/cockroach/pkg/util/allstacks.Get(...)
F231016 12:33:16.715338 96386 storage/pebble.go:1268 ⋮ [T1,n10] 1347 !	github.com/cockroachdb/cockroach/pkg/util/allstacks/allstacks.go:19
F231016 12:33:16.715338 96386 storage/pebble.go:1268 ⋮ [T1,n10] 1347 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(0xc000f1a620, {{{0xc000dfadb0, 0x24}, {0x5f96180, 0x2}, {0x5dfbbec, 0x1}, {0x5f96180, 0x2}}, 0x178e96827acbac6c, ...})
F231016 12:33:16.715338 96386 storage/pebble.go:1268 ⋮ [T1,n10] 1347 !	github.com/cockroachdb/cockroach/pkg/util/log/clog.go:263 +0x99
F231016 12:33:16.715338 96386 storage/pebble.go:1268 ⋮ [T1,n10] 1347 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x7361630, 0xc000e7ac90}, 0x2, 0x4, 0x0, 0x0?, {0x5dc4ca9, 0x1d}, {0xc003f6fdc8, 0x1, ...})
F231016 12:33:16.715338 96386 storage/pebble.go:1268 ⋮ [T1,n10] 1347 !	github.com/cockroachdb/cockroach/pkg/util/log/channels.go:106 +0x645
F231016 12:33:16.715338 96386 storage/pebble.go:1268 ⋮ [T1,n10] 1347 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepth(...)
F231016 12:33:16.715338 96386 storage/pebble.go:1268 ⋮ [T1,n10] 1347 !	github.com/cockroachdb/cockroach/pkg/util/log/channels.go:39
F231016 12:33:16.715338 96386 storage/pebble.go:1268 ⋮ [T1,n10] 1347 !github.com/cockroachdb/cockroach/pkg/util/log.Fatalf(...)
F231016 12:33:16.715338 96386 storage/pebble.go:1268 ⋮ [T1,n10] 1347 !	github.com/cockroachdb/cockroach/bazel-out/k8-opt/bin/pkg/util/log/log_channels_generated.go:848
F231016 12:33:16.715338 96386 storage/pebble.go:1268 ⋮ [T1,n10] 1347 !github.com/cockroachdb/cockroach/pkg/storage.(*Pebble).makeMetricEtcEventListener.func4({{0xc024585640, 0x3d}, 0x2, 0x0, 0x4a824bb93})
F231016 12:33:16.715338 96386 storage/pebble.go:1268 ⋮ [T1,n10] 1347 !	github.com/cockroachdb/cockroach/pkg/storage/pebble.go:1268 +0x215
F231016 12:33:16.715338 96386 storage/pebble.go:1268 ⋮ [T1,n10] 1347 !github.com/cockroachdb/pebble.TeeEventListener.func4({{0xc024585640, 0x3d}, 0x2, 0x0, 0x4a824bb93})
F231016 12:33:16.715338 96386 storage/pebble.go:1268 ⋮ [T1,n10] 1347 !	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/event.go:672 +0x4d
F231016 12:33:16.715338 96386 storage/pebble.go:1268 ⋮ [T1,n10] 1347 !github.com/cockroachdb/cockroach/pkg/storage.wrapFilesystemMiddleware.func1({{0xc024585640, 0x3d}, 0x2, 0x0, 0x4a824bb93})
F231016 12:33:16.715338 96386 storage/pebble.go:1268 ⋮ [T1,n10] 1347 !	github.com/cockroachdb/cockroach/pkg/storage/pebble.go:668 +0x97
F231016 12:33:16.715338 96386 storage/pebble.go:1268 ⋮ [T1,n10] 1347 !github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).Create.func2(0xca?, 0xe638dd42b3?, 0xb4f5580?)
F231016 12:33:16.715338 96386 storage/pebble.go:1268 ⋮ [T1,n10] 1347 !	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:633 +0x6b
F231016 12:33:16.715338 96386 storage/pebble.go:1268 ⋮ [T1,n10] 1347 !github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1()
F231016 12:33:16.715338 96386 storage/pebble.go:1268 ⋮ [T1,n10] 1347 !	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:187 +0x1cf
F231016 12:33:16.715338 96386 storage/pebble.go:1268 ⋮ [T1,n10] 1347 !created by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker
F231016 12:33:16.715338 96386 storage/pebble.go:1268 ⋮ [T1,n10] 1347 !	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:167 +0x5d
F231016 12:33:16.715338 96386 storage/pebble.go:1268 ⋮ [T1,n10] 1347 !

[jbowens]

Linking to #97968.