roachtest: schemachange/during/tpcc failed [disk stall]

[cockroach-teamcity]

roachtest.schemachange/during/tpcc failed with artifacts on master @ 174ec84d49a0e478a4052361fcbeceea62400bcc:

(monitor.go:153).Wait: monitor failure: unexpected node event: n1: cockroach process for system interface died (exit code 7)
test artifacts and logs in: /artifacts/schemachange/during/tpcc/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-32820

[rafiss]

F231027 16:17:35.356271 81773686 storage/pebble.go:1400 ⋮ [n1,s1,pebble] 8065  file write stall detected: disk slowness detected: syncto on file 058206.sst (7368704 bytes) has been ongoing for 21.8s

looks like an instance of #97968 but I will let Storage confirm that.

[jbowens]

goroutine 948507 [syscall]:
syscall.Syscall(0xc00ed5ead8?, 0x123?, 0x8000000000000000?, 0xc01d213bb0?)
	GOROOT/src/syscall/syscall_linux.go:69 +0x27
golang.org/x/sys/unix.Fdatasync(0x6c53fff998d?)
	golang.org/x/sys/unix/external/org_golang_x_sys/unix/zsyscall_linux.go:733 +0x30
github.com/cockroachdb/pebble/vfs.(*linuxFile).SyncData(0x6c53fff998d?)
	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/default_linux.go:65 +0x1d
github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).SyncData.func1()
	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:264 +0x2e
github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).timeDiskOp(0xc00e8f6ff0, 0x3, 0x0, 0xc01d213c88)
	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:297 +0x135
github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).SyncData(0x0?)
	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:263 +0x55
github.com/cockroachdb/pebble/vfs.(*enospcFile).SyncData(0x4f5eb7?)
	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_full.go:436 +0x22
github.com/cockroachdb/pebble/vfs.(*syncingFile).Sync(0xc00d002e88?)
	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/syncing_file.go:111 +0x4c
github.com/cockroachdb/pebble/record.(*LogWriter).syncWithLatency(0xc013160000)
	github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:552 +0x43
github.com/cockroachdb/pebble/record.(*LogWriter).flushPending(0xc013160000, {0xc00ed5ead8, 0x123, 0x3530}, {0
	github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:539 +0x1da
github.com/cockroachdb/pebble/record.(*LogWriter).flushLoop(0xc013160000, {0x5978680, 0xc1c8a28})
	github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:473 +0x3ba
runtime/pprof.Do({0x794dad8?, 0xc0000be088?}, {{0xc000630860?, 0xc004e4cd00?, 0xc00157c3c0?}}, 0xc006b21fc0)
	GOROOT/src/runtime/pprof/runtime.go:44 +0xa3
github.com/cockroachdb/pebble/record.NewLogWriter.func2()
	github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:355 +0x5c
created by github.com/cockroachdb/pebble/record.NewLogWriter
	github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:354 +0x416\