roachtest: clearrange/checks=true failed

[cockroach-teamcity]

(roachtest).clearrange/checks=true failed on provisional_202008031850_v20.2.0-alpha.3@b0ec3c9834d59be51e7af8a070bbef8b20f2d62a:

	cluster.go:2556,clearrange.go:170,clearrange.go:33,test_runner.go:754: monitor failure: unexpected node event: 10: dead
		(1) attached stack trace
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2544
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2552
		  | main.runClearRange
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/clearrange.go:170
		  | main.registerClearRange.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/clearrange.go:33
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:754
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1373
		Wraps: (2) monitor failure
		Wraps: (3) unexpected node event: 10: dead
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *errors.errorString

	cluster.go:1571,context.go:135,cluster.go:1560,test_runner.go:823: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2143445-1596487173-29-n10cpu4 --oneshot --ignore-empty-nodes: exit status 1 10: dead
		3: 4650
		5: 4551
		6: 4576
		7: 4524
		8: 4489
		9: 4539
		4: 4466
		1: 4852
		2: 4955
		Error: UNCLASSIFIED_PROBLEM: 10: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  | main.glob..func13
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1115
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:266
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:864
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1808
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:203
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1373
		Wraps: (3) 3 safe details enclosed
		Wraps: (4) 10: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *safedetails.withSafeDetails (4) *errors.errorString

More

Artifacts: /clearrange/checks=true
Related:

• roachtest: clearrange/checks=true failed #51211 roachtest: clearrange/checks=true failed C-test-failure O-roachtest O-robot branch-provisional_202007081918_v20.2.0-alpha.2 release-blocker

• roachtest: clearrange/checks=true failed #51128 roachtest: clearrange/checks=true failed C-test-failure O-roachtest O-robot branch-provisional_202007071743_v20.2.0-alpha.2 release-blocker

• roachtest: clearrange/checks=true failed #51021 roachtest: clearrange/checks=true failed C-test-failure O-roachtest O-robot branch-provisional_202006292108_v19.1.11 release-blocker

• roachtest: clearrange/checks=true failed #50870 roachtest: clearrange/checks=true failed C-test-failure O-roachtest O-robot branch-master release-blocker

• roachtest: clearrange/checks=true failed #48414 roachtest: clearrange/checks=true failed C-test-failure O-roachtest O-robot branch-provisional_202005041945_v19.1.9 release-blocker

• roachtest: clearrange/checks=true failed #44845 roachtest: clearrange/checks=true failed A-testing C-test-failure O-roachtest O-robot branch-release-19.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

[itsbilal]

Should be fixed by cockroachdb/pebble#869