roachtest: acceptance/gossip/restart-node-one failed

[cockroach-teamcity]

(roachtest).acceptance/gossip/restart-node-one failed on master@998abbe628d7133932c1beb9240a18c229bab735:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: artifacts/acceptance/gossip/restart-node-one/run_1
	cluster.go:1941,gossip.go:305,acceptance.go:91,test_runner.go:753: /go/src/github.com/cockroachdb/cockroach/bin/roachprod start --racks=4 --env=COCKROACH_SCAN_MAX_IDLE_TIME=5ms --encrypt=false local returned: exit status 1
		(1) /go/src/github.com/cockroachdb/cockroach/bin/roachprod start --racks=4 --env=COCKROACH_SCAN_MAX_IDLE_TIME=5ms --encrypt=false local returned
		  | stderr:
		  | ckroach/pkg/cmd/roachprod/install.(*SyncedCluster).Parallel.func1.1
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:1660
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357: 
		  | 3: invalid version string '998abbe'
		  | github.com/cockroachdb/cockroach/pkg/util/version.Parse
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/util/version/version.go:90
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.getCockroachVersion
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:96
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.Cockroach.Start.func2
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:168
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*SyncedCluster).Parallel.func1.1
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:1660
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357: 
		  | I200422 01:31:44.073679 1 cluster_synced.go:1742  command failed
		  |
		  | stdout:
		  | local: starting
		Wraps: (2) exit status 1
		Error types: (1) *main.withCommandDetails (2) *exec.ExitError

More

Artifacts: /acceptance/gossip/restart-node-one

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

[cockroach-teamcity]

(roachtest).acceptance/gossip/restart failed on master@2bcb72dec25261266f5b0bad283f0539d3ccc7b5:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: artifacts/acceptance/gossip/restart/run_1
	test_runner.go:800: test timed out (10m0s)

	gossip.go:230,gossip.go:290,acceptance.go:91,test_runner.go:753: dial tcp 127.0.0.1:26263: connect: connection refused

	cluster.go:1481,context.go:135,cluster.go:1470,test_runner.go:825: dead node detection: /go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor local --oneshot --ignore-empty-nodes: exit status 1 4: dead
		1: dead
		2: 23762
		3: 23864
		Error: UNCLASSIFIED_PROBLEM: 4: dead, 1: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) 4: dead, 1: dead
		  | main.glob..func13
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1129
		  | main.wrap.func1
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:272
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute
		  | 	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:766
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute
		  | 	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
		  | main.main
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1793
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:203
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Error types: (1) errors.Unclassified (2) *errors.fundamental

More

Artifacts: /acceptance/gossip/restart
Related:

• roachtest: acceptance/gossip/restart-node-one failed #47824 roachtest: acceptance/gossip/restart-node-one failed C-test-failure O-roachtest O-robot branch-provisional_202004210018_v20.1.0 release-blocker

• roachtest: acceptance/gossip/restart failed #47823 roachtest: acceptance/gossip/restart failed C-test-failure O-roachtest O-robot branch-provisional_202004210018_v20.1.0 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

[knz]

that's not the same failure as that fixed by #47799 so perhaps worth investigating

[irfansharif]

For some reason a failure for acceptance/gossip/restart was captured here. n1 and n4 both segfaulted.

SIGSEGV: segmentation violation
PC=0x784883 m=0 sigcode=0

goroutine 148 [syscall]:
runtime.notetsleepg(0x74cd940, 0xd8a9ec5c5, 0x0)
	/usr/local/go/src/runtime/lock_futex.go:227 +0x34 fp=0xc0004a1760 sp=0xc0004a1730 pc=0x72e094
runtime.timerproc(0x74cd920)
	/usr/local/go/src/runtime/time.go:311 +0x2f1 fp=0xc0004a17d8 sp=0xc0004a1760 pc=0x7715a1
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0004a17e0 sp=0xc0004a17d8 pc=0x782961
created by runtime.(*timersBucket).addtimerLocked
	/usr/local/go/src/runtime/time.go:169 +0x10e

SIGSEGV: segmentation violation
PC=0x784883 m=0 sigcode=0

goroutine 6 [syscall]:
runtime.notetsleepg(0x74cd5c0, 0x23c30526, 0x1)
	/usr/local/go/src/runtime/lock_futex.go:227 +0x34 fp=0xc0000a2760 sp=0xc0000a2730 pc=0x72e094
runtime.timerproc(0x74cd5a0)
	/usr/local/go/src/runtime/time.go:311 +0x2f1 fp=0xc0000a27d8 sp=0xc0000a2760 pc=0x7715a1
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0000a27e0 sp=0xc0000a27d8 pc=0x782961
created by runtime.(*timersBucket).addtimerLocked
	/usr/local/go/src/runtime/time.go:169 +0x10e

[irfansharif]

The failure from before was fixed by #47799.

[irfansharif]

Segfault being tracked in #44069 (comment).