roachtest: restore/nodeShutdown/coordinator failed #84891

cockroach-teamcity · 2022-07-22T05:27:10Z

roachtest.restore/nodeShutdown/coordinator failed with artifacts on master @ fafc3330130a016a881592d0337ec48d9ce43b61:

test artifacts and logs in: /artifacts/restore/nodeShutdown/coordinator/run_1
	monitor.go:127,jobs.go:154,restore.go:318,test_runner.go:896: monitor failure: monitor task failed: unexpectedly found job 781179773512810498 in state failed
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.jobSurvivesNodeShutdown
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/jobs.go:154
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerRestoreNodeShutdown.func3
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/restore.go:318
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:896
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.jobSurvivesNodeShutdown.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/jobs.go:95
		  | main.(*monitorImpl).Go.func1
		  | 	main/pkg/cmd/roachtest/monitor.go:105
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:74
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (6) unexpectedly found job 781179773512810498 in state failed
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/bulk-io _{This test on roachdash | Improve this report!

Jira issue: CRDB-17928}

The text was updated successfully, but these errors were encountered:

cockroach-teamcity · 2022-07-25T05:26:26Z

roachtest.restore/nodeShutdown/coordinator failed with artifacts on master @ eabdc49383a19d5731b765dbb0d8b45bd9e24404:

test artifacts and logs in: /artifacts/restore/nodeShutdown/coordinator/run_1
	monitor.go:127,jobs.go:154,restore.go:318,test_runner.go:896: monitor failure: monitor task failed: unexpectedly found job 782029002816487426 in state failed
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.jobSurvivesNodeShutdown
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/jobs.go:154
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerRestoreNodeShutdown.func3
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/restore.go:318
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:896
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.jobSurvivesNodeShutdown.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/jobs.go:95
		  | main.(*monitorImpl).Go.func1
		  | 	main/pkg/cmd/roachtest/monitor.go:105
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:74
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (6) unexpectedly found job 782029002816487426 in state failed
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-07-28T05:27:01Z

roachtest.restore/nodeShutdown/coordinator failed with artifacts on master @ 7c95ccd0a9f0b0aca9cdb0ed813caf8d60b28732:

test artifacts and logs in: /artifacts/restore/nodeShutdown/coordinator/run_1
	monitor.go:127,jobs.go:154,restore.go:318,test_runner.go:896: monitor failure: monitor task failed: unexpectedly found job 782878287079145474 in state failed
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.jobSurvivesNodeShutdown
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/jobs.go:154
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerRestoreNodeShutdown.func3
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/restore.go:318
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:896
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.jobSurvivesNodeShutdown.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/jobs.go:95
		  | main.(*monitorImpl).Go.func1
		  | 	main/pkg/cmd/roachtest/monitor.go:105
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:74
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (6) unexpectedly found job 782878287079145474 in state failed
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

rhu713 · 2022-07-29T16:35:44Z

RESTORE job 781179773512810498: stepping through state failed with error: importing 41 ranges: splitting key ‹/Table/106/1/8759097›: refusing to transfer lease to (n2,s2):5VOTER_INCOMING because target may need a Raft snapshot: ‹replica in StateProbe›

From slack:
hardened some checks around transferring leases to nodes that will then be stalled because they can’t be the leaseholder (they’re missing some of the state). Seems like a temporary error, is the action not retried ? or does the test fail before its retried ?

Seems to be similar line of temp errors that need to be retried on AdminSplit(), removing release blocker.

msbutler · 2022-07-29T16:42:24Z

^ thanks for digging! I'll open a fix for admin split retries.

cockroach-teamcity · 2022-09-25T15:29:13Z

roachtest.restore/nodeShutdown/coordinator failed with artifacts on master @ 51c8aae748d338549400c047796c6c9b892527da:

test artifacts and logs in: /artifacts/restore/nodeShutdown/coordinator/run_1
	monitor.go:127,jobs.go:154,restore.go:318,test_runner.go:928: monitor failure: monitor command failure: unexpected node event: 3: dead (exit status 137)
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.jobSurvivesNodeShutdown
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/jobs.go:154
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerRestoreNodeShutdown.func3
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/restore.go:318
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func3
		  | 	main/pkg/cmd/roachtest/monitor.go:202
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (4) monitor command failure
		Wraps: (5) unexpected node event: 3: dead (exit status 137)
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *errors.errorString

	test_runner.go:1059,test_runner.go:958: test timed out (0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: restore/nodeShutdown/coordinator failed #85879 roachtest: restore/nodeShutdown/coordinator failed [C-test-failure O-roachtest O-robot T-kv branch-release-22.1]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-09-27T15:28:43Z

roachtest.restore/nodeShutdown/coordinator failed with artifacts on master @ a0bfa6dafcc206301d3a21887c374db63b377075:

test artifacts and logs in: /artifacts/restore/nodeShutdown/coordinator/run_1
	monitor.go:127,jobs.go:154,restore.go:318,test_runner.go:928: monitor failure: monitor command failure: unexpected node event: 1: dead (exit status 137)
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.jobSurvivesNodeShutdown
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/jobs.go:154
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerRestoreNodeShutdown.func3
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/restore.go:318
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func3
		  | 	main/pkg/cmd/roachtest/monitor.go:202
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (4) monitor command failure
		Wraps: (5) unexpected node event: 1: dead (exit status 137)
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *errors.errorString

	test_runner.go:1059,test_runner.go:958: test timed out (0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: restore/nodeShutdown/coordinator failed #85879 roachtest: restore/nodeShutdown/coordinator failed [C-test-failure O-roachtest O-robot T-kv branch-release-22.1]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-09-28T15:30:47Z

roachtest.restore/nodeShutdown/coordinator failed with artifacts on master @ 4982322e91c37c6d0249dc5597b9e02eec19848d:

test artifacts and logs in: /artifacts/restore/nodeShutdown/coordinator/run_1
	monitor.go:127,jobs.go:154,restore.go:318,test_runner.go:928: monitor failure: monitor command failure: unexpected node event: 4: dead (exit status 137)
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.jobSurvivesNodeShutdown
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/jobs.go:154
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerRestoreNodeShutdown.func3
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/restore.go:318
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func3
		  | 	main/pkg/cmd/roachtest/monitor.go:202
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (4) monitor command failure
		Wraps: (5) unexpected node event: 4: dead (exit status 137)
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *errors.errorString

	test_runner.go:1059,test_runner.go:958: test timed out (0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: restore/nodeShutdown/coordinator failed #85879 roachtest: restore/nodeShutdown/coordinator failed [C-test-failure O-roachtest O-robot T-kv branch-release-22.1]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-09-30T05:18:06Z

roachtest.restore/nodeShutdown/coordinator failed with artifacts on master @ 4780ee15194e7d22b661ac92254b13cd2b71dcb1:

test artifacts and logs in: /artifacts/restore/nodeShutdown/coordinator/run_1
	monitor.go:127,jobs.go:154,restore.go:318,test_runner.go:928: monitor failure: monitor command failure: unexpected node event: 4: dead (exit status 137)
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.jobSurvivesNodeShutdown
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/jobs.go:154
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerRestoreNodeShutdown.func3
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/restore.go:318
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func3
		  | 	main/pkg/cmd/roachtest/monitor.go:202
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (4) monitor command failure
		Wraps: (5) unexpected node event: 4: dead (exit status 137)
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *errors.errorString

	test_runner.go:1059,test_runner.go:958: test timed out (0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: restore/nodeShutdown/coordinator failed #85879 roachtest: restore/nodeShutdown/coordinator failed [C-test-failure O-roachtest O-robot T-kv branch-release-22.1]

_{This test on roachdash | Improve this report!}

adityamaru · 2023-06-26T17:20:46Z

This hasn't failed in a while, closing

cockroach-teamcity added this to the 22.2 milestone Jul 22, 2022

blathers-crl bot added the T-disaster-recovery label Jul 22, 2022

shermanCRL assigned rhu713 Jul 25, 2022

rhu713 removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Jul 29, 2022

rhu713 mentioned this issue Jul 29, 2022

backupccl: retry restore's AdminSplit on ephemeral errors #85333

Closed

cockroach-teamcity mentioned this issue Aug 10, 2022

roachtest: restore/nodeShutdown/coordinator failed #85879

Closed

adityamaru closed this as completed Jun 26, 2023

github-project-automation bot added this to Disaster Recovery Backlog Aug 28, 2024

github-project-automation bot moved this to Done in Disaster Recovery Backlog Aug 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

roachtest: restore/nodeShutdown/coordinator failed #84891

roachtest: restore/nodeShutdown/coordinator failed #84891

cockroach-teamcity commented Jul 22, 2022 •

edited by cockroach-jira-scripts

Loading

cockroach-teamcity commented Jul 25, 2022

cockroach-teamcity commented Jul 28, 2022

rhu713 commented Jul 29, 2022

msbutler commented Jul 29, 2022

cockroach-teamcity commented Sep 25, 2022

cockroach-teamcity commented Sep 27, 2022

cockroach-teamcity commented Sep 28, 2022

cockroach-teamcity commented Sep 30, 2022

adityamaru commented Jun 26, 2023

roachtest: restore/nodeShutdown/coordinator failed #84891

roachtest: restore/nodeShutdown/coordinator failed #84891

Comments

cockroach-teamcity commented Jul 22, 2022 • edited by cockroach-jira-scripts Loading

cockroach-teamcity commented Jul 25, 2022

cockroach-teamcity commented Jul 28, 2022

rhu713 commented Jul 29, 2022

msbutler commented Jul 29, 2022

cockroach-teamcity commented Sep 25, 2022

cockroach-teamcity commented Sep 27, 2022

cockroach-teamcity commented Sep 28, 2022

cockroach-teamcity commented Sep 30, 2022

adityamaru commented Jun 26, 2023

cockroach-teamcity commented Jul 22, 2022 •

edited by cockroach-jira-scripts

Loading