roachtest: clearrange/checks=false failed

[cockroach-teamcity]

(roachtest).clearrange/checks=false failed on provisional_202007220233_v20.2.0-alpha.2@d3119926d33d808c6384cf3e99a7f7435f395489:

	cluster.go:2553,clearrange.go:170,clearrange.go:33,test_runner.go:757: monitor failure: unexpected node event: 5: dead
		(1) attached stack trace
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2541
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2549
		  | main.runClearRange
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/clearrange.go:170
		  | main.registerClearRange.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/clearrange.go:33
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:757
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1373
		Wraps: (2) monitor failure
		Wraps: (3) unexpected node event: 5: dead
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *errors.errorString

	cluster.go:1571,context.go:135,cluster.go:1560,test_runner.go:826: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2107811-1595392378-25-n10cpu4 --oneshot --ignore-empty-nodes: exit status 1 4: 4946
		3: 4438
		5: dead
		7: 4555
		1: 4746
		2: 4433
		10: 4547
		8: 4477
		9: 4672
		6: 4661
		Error: UNCLASSIFIED_PROBLEM: 5: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  | main.glob..func13
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1115
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:266
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:864
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1808
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:203
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1373
		Wraps: (3) 3 safe details enclosed
		Wraps: (4) 5: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *safedetails.withSafeDetails (4) *errors.errorString

More

Artifacts: /clearrange/checks=false
Related:

• roachtest: clearrange/checks=false failed #51020 roachtest: clearrange/checks=false failed C-test-failure O-roachtest O-robot branch-provisional_202006292108_v19.1.11 release-blocker

• roachtest: clearrange/checks=false failed #48415 roachtest: clearrange/checks=false failed C-test-failure O-roachtest O-robot branch-provisional_202005041945_v19.1.9 release-blocker

• roachtest: clearrange/checks=false failed #41123 roachtest: clearrange/checks=false failed C-test-failure O-roachtest O-robot branch-master

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

[irfansharif]

F200722 06:36:47.291829 29 server/server.go:300  [n3] clock synchronization error: this node is more than 500ms away from at least half of the known nodes (5 of 11 are within the offset)
goroutine 29 [running]:

Infra-flake.