roachtest: tpccbench/nodes=9/cpu=4/multi-region failed

[cockroach-teamcity]

(roachtest).tpccbench/nodes=9/cpu=4/multi-region failed on master@48fbbe7313fc70f889fa66c5749914f57ddd5a91:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
	cluster.go:2687,tpcc.go:765,tpcc.go:600,test_runner.go:767: monitor failure: unexpected node event: 5: dead
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2675
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2683
		  | main.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:765
		  | main.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:600
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:767
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) monitor failure
		Wraps: (3) unexpected node event: 5: dead
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

More

Artifacts: /tpccbench/nodes=9/cpu=4/multi-region
Related:

• roachtest: tpccbench/nodes=9/cpu=4/multi-region failed [congmgr assertion failed] #53540 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed [congmgr assertion failed] C-test-failure O-roachtest O-robot

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

[nvanbenschoten]

F210125 14:31:15.955828 6653149 1@util/log/file.go:250 â‹® disk stall detected: unable to sync log files within 1m0s
goroutine 6653149 [running]:
github.com/cockroachdb/cockroach/pkg/util/log.getStacks(0x82c4901, 0x203087, 0x203087, 0x8)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/get_stacks.go:25 +0xb9
github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(0xc0011fe5a0, 0xc001b1a060, 0x24, 0x4, 0x0, 0x0, 0x0, 0x165d801689ab0834, 0x400000000, 0x1, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:279 +0xc32
github.com/cockroachdb/cockroach/pkg/util/log.logfDepth(0x565b480, 0xc00007a0d8, 0x2, 0x100000004, 0x4a20171, 0x37, 0xc001955fc0, 0x1, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/channels.go:58 +0x198
github.com/cockroachdb/cockroach/pkg/util/log.shoutfDepth(0x565b480, 0xc00007a0d8, 0x1, 0x100000004, 0x4a20171, 0x37, 0xc017cdefc0, 0x1, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/channels.go:82 +0xdb
github.com/cockroachdb/cockroach/pkg/util/log.loggerOps.Shoutf(...)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log_channels_generated.go:1385
github.com/cockroachdb/cockroach/pkg/util/log.(*fileSink).flushAndMaybeSyncLocked.func1()
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/file.go:250 +0xbe
created by time.goFunc
	/usr/local/go/src/time/sleep.go:167 +0x45

Likely fixed by #59370, according to @tbg in #59190.