roachtest: restore/nodeShutdown/worker failed #84148

cockroach-teamcity · 2022-07-10T05:25:41Z

roachtest.restore/nodeShutdown/worker failed with artifacts on master @ 6374bd83f192bd509e50313acb431690b9c3f8ae:

test artifacts and logs in: /artifacts/restore/nodeShutdown/worker/run_1
	monitor.go:127,jobs.go:154,restore.go:304,test_runner.go:896: monitor failure: monitor task failed: unexpectedly found job 777782192977215490 in state failed
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.jobSurvivesNodeShutdown
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/jobs.go:154
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerRestoreNodeShutdown.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/restore.go:304
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:896
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.jobSurvivesNodeShutdown.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/jobs.go:95
		  | main.(*monitorImpl).Go.func1
		  | 	main/pkg/cmd/roachtest/monitor.go:105
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:74
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (6) unexpectedly found job 777782192977215490 in state failed
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: restore/nodeShutdown/worker failed #80821 roachtest: restore/nodeShutdown/worker failed [C-test-failure O-roachtest O-robot T-bulkio branch-release-21.1]

/cc @cockroachdb/bulk-io _{This test on roachdash | Improve this report!

Jira issue: CRDB-17491}

The text was updated successfully, but these errors were encountered:

msbutler · 2022-07-11T17:22:03Z

While node 3 was shutting down, the restore job failed with the following message, according to the jobs table:
importing 41 ranges: splitting key /Table/106/1/369393 3: refusing to transfer lease to (n3,s3):4VOTER_INCOMING because target may need a Raft snapshot: replica in StateProbe.

Node 2 coordinated the job and its log surfaced the same errors:

2.unredacted/cockroach.log:I220710 05:25:20.612291 2814 jobs/registry.go:1208 ⋮ [n2] 333  RESTORE job 777782192977215490: stepping through state reverting with error: importing 41 ranges: splitting key ‹/Table/106/1/3693933›: refusing to transfer lease to (n3,s3):4VOTER_INCOMING because target may need a Raft snapshot: ‹replica in StateProbe›
2.unredacted/cockroach.log:I220710 05:25:20.612291 2814 jobs/registry.go:1208 ⋮ [n2] 333 +Wraps: (9) refusing to transfer lease to (n3,s3):4VOTER_INCOMING because target may need a Raft snapshot: ‹replica in StateProbe›
2.unredacted/cockroach.log:I220710 05:25:20.645371 2814 jobs/registry.go:1208 ⋮ [n2] 336  RESTORE job 777782192977215490: stepping through state failed with error: importing 41 ranges: splitting key ‹/Table/106/1/3693933›:refusing to transfer lease to (n3,s3):4VOTER_INCOMING because target may need a Raft snapshot: ‹replica in StateProbe›
2.unredacted/cockroach.log:I220710 05:25:20.645371 2814 jobs/registry.go:1208 ⋮ [n2] 336 +Wraps: (9) refusing to transfer lease to (n3,s3):4VOTER_INCOMING because target may need a Raft snapshot: ‹replica in StateProbe›
2.unredacted/cockroach.log:E220710 05:25:20.649694 2814 jobs/adopt.go:421 ⋮ [n2] 337  job 777782192977215490: adoption completed with error importing 41 ranges: splitting key ‹/Table/106/1/3693933›: refusing to transfer lease to (n3,s3):4VOTER_INCOMING because target may need a Raft snapshot: ‹replica in StateProbe›
➜  logs

I think this failure is caused by the node shutting down, and independent of the restore job, so I'm removing bulk from the issue and handing to KV for triage. I bet it's fallout from #83261

cockroach-teamcity · 2022-07-12T05:27:00Z

roachtest.restore/nodeShutdown/worker failed with artifacts on master @ 571bfa3afb3858ae84d8a8fcdbb0a38e058402a5:

test artifacts and logs in: /artifacts/restore/nodeShutdown/worker/run_1
	monitor.go:127,jobs.go:154,restore.go:304,test_runner.go:896: monitor failure: monitor command failure: unexpected node event: 4: dead (exit status 134)
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.jobSurvivesNodeShutdown
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/jobs.go:154
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerRestoreNodeShutdown.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/restore.go:304
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func3
		  | 	main/pkg/cmd/roachtest/monitor.go:202
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor command failure
		Wraps: (5) unexpected node event: 4: dead (exit status 134)
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *errors.errorString

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: restore/nodeShutdown/worker failed #80821 roachtest: restore/nodeShutdown/worker failed [C-test-failure O-roachtest O-robot T-bulkio branch-release-21.1]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-07-13T05:26:27Z

roachtest.restore/nodeShutdown/worker failed with artifacts on master @ 687171ac6c2cd9992486bb3b8c9d252ac95ca1cd:

test artifacts and logs in: /artifacts/restore/nodeShutdown/worker/run_1
	monitor.go:127,jobs.go:154,restore.go:304,test_runner.go:896: monitor failure: monitor command failure: unexpected node event: 2: dead (exit status 134)
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.jobSurvivesNodeShutdown
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/jobs.go:154
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerRestoreNodeShutdown.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/restore.go:304
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func3
		  | 	main/pkg/cmd/roachtest/monitor.go:202
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor command failure
		Wraps: (5) unexpected node event: 2: dead (exit status 134)
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *errors.errorString

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: restore/nodeShutdown/worker failed #80821 roachtest: restore/nodeShutdown/worker failed [C-test-failure O-roachtest O-robot T-bulkio branch-release-21.1]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-07-14T05:40:35Z

roachtest.restore/nodeShutdown/worker failed with artifacts on master @ 88d3253301457ac57820e0f4a4fab8f74bf9f38b:

test artifacts and logs in: /artifacts/restore/nodeShutdown/worker/run_1
	monitor.go:127,jobs.go:154,restore.go:304,test_runner.go:896: monitor failure: monitor command failure: unexpected node event: 2: dead (exit status 134)
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.jobSurvivesNodeShutdown
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/jobs.go:154
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerRestoreNodeShutdown.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/restore.go:304
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func3
		  | 	main/pkg/cmd/roachtest/monitor.go:202
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) monitor command failure
		Wraps: (5) unexpected node event: 2: dead (exit status 134)
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *errors.errorString

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: restore/nodeShutdown/worker failed #80821 roachtest: restore/nodeShutdown/worker failed [C-test-failure O-roachtest O-robot T-bulkio branch-release-21.1]

_{This test on roachdash | Improve this report!}

lidorcarmel · 2022-07-26T23:52:22Z

@nvanbenschoten is it related to #83261 as Michael said above? not clear to me why this stopped failing.
Also, is it a release blocker?

cockroach-teamcity · 2022-07-30T05:26:21Z

roachtest.restore/nodeShutdown/worker failed with artifacts on master @ cd113bf428e8868c38b2f2f295210f5c30d13a2d:

test artifacts and logs in: /artifacts/restore/nodeShutdown/worker/run_1
	monitor.go:127,jobs.go:154,restore.go:304,test_runner.go:896: monitor failure: monitor task failed: unexpectedly found job 783444657148166146 in state failed
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.jobSurvivesNodeShutdown
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/jobs.go:154
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerRestoreNodeShutdown.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/restore.go:304
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:896
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.jobSurvivesNodeShutdown.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/jobs.go:95
		  | main.(*monitorImpl).Go.func1
		  | 	main/pkg/cmd/roachtest/monitor.go:105
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:74
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (6) unexpectedly found job 783444657148166146 in state failed
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: (triaged) restore/nodeShutdown/worker failed [from transient admin split error] #84635 roachtest: restore/nodeShutdown/worker failed [C-test-failure O-roachtest O-robot T-bulkio branch-release-22.1]
roachtest: restore/nodeShutdown/worker failed #80821 roachtest: restore/nodeShutdown/worker failed [C-test-failure O-roachtest O-robot T-bulkio branch-release-21.1]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-08-03T05:25:49Z

roachtest.restore/nodeShutdown/worker failed with artifacts on master @ 7326bbbf94610c40ae2d2fdf42c119dfea93a170:

test artifacts and logs in: /artifacts/restore/nodeShutdown/worker/run_1
	monitor.go:127,jobs.go:154,restore.go:304,test_runner.go:896: monitor failure: monitor task failed: unexpectedly found job 784577026719514626 in state failed
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.jobSurvivesNodeShutdown
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/jobs.go:154
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerRestoreNodeShutdown.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/restore.go:304
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:896
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.jobSurvivesNodeShutdown.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/jobs.go:95
		  | main.(*monitorImpl).Go.func1
		  | 	main/pkg/cmd/roachtest/monitor.go:105
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:74
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (6) unexpectedly found job 784577026719514626 in state failed
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: (triaged) restore/nodeShutdown/worker failed [from transient admin split error] #84635 roachtest: restore/nodeShutdown/worker failed [C-test-failure O-roachtest O-robot T-bulkio branch-release-22.1]
roachtest: restore/nodeShutdown/worker failed #80821 roachtest: restore/nodeShutdown/worker failed [C-test-failure O-roachtest O-robot T-bulkio branch-release-21.1]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-08-20T05:26:08Z

roachtest.restore/nodeShutdown/worker failed with artifacts on master @ cb55144cdec54d2a70f074ad64b4eca5e6c6891a:

test artifacts and logs in: /artifacts/restore/nodeShutdown/worker/run_1
	monitor.go:127,jobs.go:154,restore.go:304,test_runner.go:896: monitor failure: monitor task failed: unexpectedly found job 789390067084623874 in state failed
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.jobSurvivesNodeShutdown
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/jobs.go:154
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerRestoreNodeShutdown.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/restore.go:304
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:896
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.jobSurvivesNodeShutdown.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/jobs.go:95
		  | main.(*monitorImpl).Go.func1
		  | 	main/pkg/cmd/roachtest/monitor.go:105
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:74
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (6) unexpectedly found job 789390067084623874 in state failed
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: (triaged) restore/nodeShutdown/worker failed [from transient admin split error] #84635 roachtest: restore/nodeShutdown/worker failed [C-test-failure O-roachtest O-robot T-bulkio branch-release-22.1 stability-period-v22.2]
roachtest: restore/nodeShutdown/worker failed #80821 roachtest: restore/nodeShutdown/worker failed [C-test-failure O-roachtest O-robot T-bulkio branch-release-21.1]

_{This test on roachdash | Improve this report!}

stevendanna · 2022-09-06T15:14:28Z

We think this is likely fixed by #85405. I'm going to close as it hasn't happened in some time.

cockroach-teamcity added this to the 22.2 milestone Jul 10, 2022

blathers-crl bot added the T-disaster-recovery label Jul 10, 2022

msbutler added the T-kv KV Team label Jul 11, 2022

msbutler removed the T-disaster-recovery label Jul 11, 2022

cockroach-teamcity mentioned this issue Jul 19, 2022

roachtest: (triaged) restore/nodeShutdown/worker failed [from transient admin split error] #84635

Closed

stevendanna closed this as completed Sep 6, 2022

github-project-automation bot added this to Disaster Recovery Backlog Aug 28, 2024

github-project-automation bot moved this to Done in Disaster Recovery Backlog Aug 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

roachtest: restore/nodeShutdown/worker failed #84148

roachtest: restore/nodeShutdown/worker failed #84148

cockroach-teamcity commented Jul 10, 2022 •

edited by cockroach-jira-scripts

Loading

msbutler commented Jul 11, 2022

cockroach-teamcity commented Jul 12, 2022

cockroach-teamcity commented Jul 13, 2022

cockroach-teamcity commented Jul 14, 2022

lidorcarmel commented Jul 26, 2022

cockroach-teamcity commented Jul 30, 2022

cockroach-teamcity commented Aug 3, 2022

cockroach-teamcity commented Aug 20, 2022

stevendanna commented Sep 6, 2022

roachtest: restore/nodeShutdown/worker failed #84148

roachtest: restore/nodeShutdown/worker failed #84148

Comments

cockroach-teamcity commented Jul 10, 2022 • edited by cockroach-jira-scripts Loading

msbutler commented Jul 11, 2022

cockroach-teamcity commented Jul 12, 2022

cockroach-teamcity commented Jul 13, 2022

cockroach-teamcity commented Jul 14, 2022

lidorcarmel commented Jul 26, 2022

cockroach-teamcity commented Jul 30, 2022

cockroach-teamcity commented Aug 3, 2022

cockroach-teamcity commented Aug 20, 2022

stevendanna commented Sep 6, 2022

cockroach-teamcity commented Jul 10, 2022 •

edited by cockroach-jira-scripts

Loading