roachtest: restart/down-for-2m failed

[cockroach-teamcity]

(roachtest).restart/down-for-2m failed on provisional_202003181957_v20.1.0-beta.3@37f2a6964760b5d72bef655342628e31601b8978:

The test failed on branch=provisional_202003181957_v20.1.0-beta.3, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200319-1816942/restart/down-for-2m/run_1
	test_runner.go:800: test timed out (10h0m0s)

	restart.go:76,restart.go:93,test_runner.go:753: dial tcp 104.154.141.228:26257: connect: connection refused

	cluster.go:1410,context.go:135,cluster.go:1399,test_runner.go:825: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1816942-1584592838-19-n3cpu4 --oneshot --ignore-empty-nodes: exit status 1 1: 23211
		3: dead
		2: 29888
		Error:  3: dead

More

Artifacts: /restart/down-for-2m
Related:

• roachtest: restart/down-for-2m failed #44060 roachtest: restart/down-for-2m failed C-test-failure O-roachtest O-robot branch-master

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

[nvanbenschoten]

SIGSEGV: segmentation violation
PC=0x7cc9c3 m=0 sigcode=0

goroutine 18 [syscall]:
runtime.notetsleepg(0x74bc620, 0x5f41bfa, 0x0)
	/usr/local/go/src/runtime/lock_futex.go:227 +0x34 fp=0xc00008c760 sp=0xc00008c730 pc=0x7761c4
runtime.timerproc(0x74bc600)
	/usr/local/go/src/runtime/time.go:311 +0x2f1 fp=0xc00008c7d8 sp=0xc00008c760 pc=0x7b96e1
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00008c7e0 sp=0xc00008c7d8 pc=0x7caaa1
created by runtime.(*timersBucket).addtimerLocked
	/usr/local/go/src/runtime/time.go:169 +0x10e

goroutine 1 [select, 584 minutes]:
runtime.gopark(0x44811a8, 0x0, 0x1809, 0x1)
	/usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0xc00354d408 sp=0xc00354d3e8 pc=0x79c550
runtime.selectgo(0xc00354d9f8, 0xc00354d680, 0x4, 0x74ff9b8, 0x4133320)
	/usr/local/go/src/runtime/select.go:313 +0xc9b fp=0xc00354d530 sp=0xc00354d408 pc=0x7ac10b
github.com/cockroachdb/cockroach/pkg/cli.runStart(0x6e1b320, 0xc000c1e500, 0x0, 0xa, 0x42ca000, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/cli/start.go:880 +0xc7d fp=0xc00354daa8 sp=0xc00354d530 pc=0x319a3bd
github.com/cockroachdb/cockroach/pkg/cli.runStartJoin(0x6e1b320, 0xc000c1e500, 0x0, 0xa, 0xc00064c900, 0xc000369a20)
	/go/src/github.com/cockroachdb/cockroach/pkg/cli/start.go:445 +0x4e fp=0xc00354daf0 sp=0xc00354daa8 pc=0x319970e
github.com/cockroachdb/cockroach/pkg/cli.MaybeDecorateGRPCError.func1(0x6e1b320, 0xc000c1e500, 0x0, 0xa, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/cli/error.go:187 +0x79 fp=0xc00354dcc0 sp=0xc00354daf0 pc=0x31af699
github.com/cockroachdb/cockroach/pkg/cli.maybeShoutError.func1(0x6e1b320, 0xc000c1e500, 0x0, 0xa, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/cli/error.go:351 +0x51 fp=0xc00354dd00 sp=0xc00354dcc0 pc=0x31b0661
github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute(0x6e1b320, 0xc000c1e3c0, 0xa, 0xa, 0x6e1b320, 0xc000c1e3c0)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:762 +0x460 fp=0xc00354ddd8 sp=0xc00354dd00 pc=0x2e9d9c0
github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x6e13a00, 0xc00007a110, 0x42c601c, 0x6)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852 +0x2ea fp=0xc00354dea8 sp=0xc00354ddd8 pc=0x2e9e3fa
github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute(...)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
github.com/cockroachdb/cockroach/pkg/cli.Run(...)
	/go/src/github.com/cockroachdb/cockroach/pkg/cli/cli.go:225
github.com/cockroachdb/cockroach/pkg/cli.Main()
	/go/src/github.com/cockroachdb/cockroach/pkg/cli/cli.go:67 +0x26b fp=0xc00354df50 sp=0xc00354dea8 pc=0x31549db
main.main()
	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/cockroach/main.go:26 +0x20 fp=0xc00354df60 sp=0xc00354df50 pc=0x3280e30
runtime.main()
	/usr/local/go/src/runtime/proc.go:203 +0x21e fp=0xc00354dfe0 sp=0xc00354df60 pc=0x79c17e
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00354dfe8 sp=0xc00354dfe0 pc=0x7caaa1

This is the same issue as #44069 (comment) and #41584 (comment).