roachtest: autoupgrade failed #86520

cockroach-teamcity · 2022-08-21T06:19:27Z

roachtest.autoupgrade failed with artifacts on master @ e6a7dc2f8ee39549e186bd05626c4c375b76fd04:

		  | 3	true	43	true	decommissioning	false
		  | id	is_live	replicas	is_decommissioning	membership	is_draining
		  | 3	true	41	true	decommissioning	false
		  | id	is_live	replicas	is_decommissioning	membership	is_draining
		  | 3	true	38	true	decommissioning	false
		  | id	is_live	replicas	is_decommissioning	membership	is_draining
		  | 3	true	35	true	decommissioning	false
		  | id	is_live	replicas	is_decommissioning	membership	is_draining
		  | 3	true	32	true	decommissioning	false
		  | id	is_live	replicas	is_decommissioning	membership	is_draining
		  | 3	true	29	true	decommissioning	false
		  | id	is_live	replicas	is_decommissioning	membership	is_draining
		  | 3	true	27	true	decommissioning	false
		  | id	is_live	replicas	is_decommissioning	membership	is_draining
		  | 3	true	26	true	decommissioning	false
		  | id	is_live	replicas	is_decommissioning	membership	is_draining
		  | 3	true	24	true	decommissioning	false
		  | id	is_live	replicas	is_decommissioning	membership	is_draining
		  | 3	true	23	true	decommissioning	false
		  | id	is_live	replicas	is_decommissioning	membership	is_draining
		  | 3	true	21	true	decommissioning	false
		  | id	is_live	replicas	is_decommissioning	membership	is_draining
		  | 3	true	19	true	decommissioning	false
		  | id	is_live	replicas	is_decommissioning	membership	is_draining
		  | 3	true	18	true	decommissioning	false
		  | id	is_live	replicas	is_decommissioning	membership	is_draining
		  | 3	true	15	true	decommissioning	false
		  | id	is_live	replicas	is_decommissioning	membership	is_draining
		  | 3	true	13	true	decommissioning	false
		  | id	is_live	replicas	is_decommissioning	membership	is_draining
		  | 3	true	12	true	decommissioning	false
		  | id	is_live	replicas	is_decommissioning	membership	is_draining
		  | 3	true	7	true	decommissioning	false
		  | id	is_live	replicas	is_decommissioning	membership	is_draining
		  | 3	true	6	true	decommissioning	false
		  | id	is_live	replicas	is_decommissioning	membership	is_draining
		  | 3	true	5	true	decommissioning	false
		  | id	is_live	replicas	is_decommissioning	membership	is_draining
		  | 3	true	4	true	decommissioning	false
		  | id	is_live	replicas	is_decommissioning	membership	is_draining
		  | 3	true	3	true	decommissioning	false
		  | id	is_live	replicas	is_decommissioning	membership	is_draining
		  | 3	true	2	true	decommissioning	false
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 3. Command with error:
		  | ``````
		  | ./cockroach node decommission 3 --insecure --port={pgport:3}
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/kv-triage _{This test on roachdash | Improve this report!

Jira issue: CRDB-18792
Epic CRDB-19172}

The text was updated successfully, but these errors were encountered:

cockroach-teamcity · 2022-08-23T06:13:07Z

roachtest.autoupgrade failed with artifacts on master @ 80c274877a917580af62be6eb0cd48c8c7ae9c08:

test artifacts and logs in: /artifacts/autoupgrade/run_1
	autoupgrade.go:188,autoupgrade.go:262,test_runner.go:896: pq: result is ambiguous: replica unavailable: (n5,s5):4 unable to serve request to r47:/Table/{6-8} [(n5,s5):4, (n4,s4):5, (n2,s2):3, next=6, gen=18]: lost quorum (down: (n4,s4):5,(n2,s2):3); closed timestamp: 1661235039.101590008,0 (2022-08-23 06:10:39); raft status: {"id":"4","term":11,"vote":"4","commit":50,"lead":"4","raftState":"StateLeader","applied":50,"progress":{"3":{"match":50,"next":51,"state":"StateProbe"},"4":{"match":93,"next":94,"state":"StateReplicate"},"5":{"match":0,"next":37,"state":"StateProbe"}},"leadtransferee":"0"}: have been waiting 61.80s for slow proposal Put [/Table/6/1/"cluster.preserve_downgrade_option"/0,/Min), [txn: 5ff399ca], [can-forward-ts]

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-08-24T06:20:23Z

roachtest.autoupgrade failed with artifacts on master @ 003c0360de8b64319b5f0f127b99be91dbdca8a3:

		  | 3	true	35	true	decommissioning	false
		  | id	is_live	replicas	is_decommissioning	membership	is_draining
		  | 3	true	32	true	decommissioning	false
		  | id	is_live	replicas	is_decommissioning	membership	is_draining
		  | 3	true	31	true	decommissioning	false
		  | id	is_live	replicas	is_decommissioning	membership	is_draining
		  | 3	true	29	true	decommissioning	false
		  | id	is_live	replicas	is_decommissioning	membership	is_draining
		  | 3	true	27	true	decommissioning	false
		  | id	is_live	replicas	is_decommissioning	membership	is_draining
		  | 3	true	25	true	decommissioning	false
		  | id	is_live	replicas	is_decommissioning	membership	is_draining
		  | 3	true	23	true	decommissioning	false
		  | id	is_live	replicas	is_decommissioning	membership	is_draining
		  | 3	true	21	true	decommissioning	false
		  | id	is_live	replicas	is_decommissioning	membership	is_draining
		  | 3	true	20	true	decommissioning	false
		  | id	is_live	replicas	is_decommissioning	membership	is_draining
		  | 3	true	19	true	decommissioning	false
		  | id	is_live	replicas	is_decommissioning	membership	is_draining
		  | 3	true	17	true	decommissioning	false
		  | id	is_live	replicas	is_decommissioning	membership	is_draining
		  | 3	true	16	true	decommissioning	false
		  | id	is_live	replicas	is_decommissioning	membership	is_draining
		  | 3	true	15	true	decommissioning	false
		  | id	is_live	replicas	is_decommissioning	membership	is_draining
		  | 3	true	13	true	decommissioning	false
		  | id	is_live	replicas	is_decommissioning	membership	is_draining
		  | 3	true	11	true	decommissioning	false
		  | id	is_live	replicas	is_decommissioning	membership	is_draining
		  | 3	true	9	true	decommissioning	false
		  | id	is_live	replicas	is_decommissioning	membership	is_draining
		  | 3	true	7	true	decommissioning	false
		  | id	is_live	replicas	is_decommissioning	membership	is_draining
		  | 3	true	6	true	decommissioning	false
		  | id	is_live	replicas	is_decommissioning	membership	is_draining
		  | 3	true	5	true	decommissioning	false
		  | id	is_live	replicas	is_decommissioning	membership	is_draining
		  | 3	true	4	true	decommissioning	false
		  | id	is_live	replicas	is_decommissioning	membership	is_draining
		  | 3	true	3	true	decommissioning	false
		  | id	is_live	replicas	is_decommissioning	membership	is_draining
		  | 3	true	2	true	decommissioning	false
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 3. Command with error:
		  | ``````
		  | ./cockroach node decommission 3 --insecure --port={pgport:3}
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

aayushshah15 · 2022-09-06T14:47:46Z

Quick update on this:

GCE_PROJECT=andrei-jepsen ./pkg/cmd/roachtest/roachstress.sh -uc100 -b autoupgrade -- --cpu-quota 1300 --debug

passed with no failures. I'll spin up more runs now but I suspect some of our recent decommissioning changes (#85640 and #86701) have made this roachtest faster and perhaps hidden the regression that was surfaced in the above failures.

cockroach-teamcity · 2022-09-09T06:03:33Z

roachtest.autoupgrade failed with artifacts on master @ a82711442c65cf14489c55041b45b11a1e38415b:

test artifacts and logs in: /artifacts/autoupgrade/run_1
	cluster.go:1823,autoupgrade.go:129,autoupgrade.go:262,test_runner.go:906: one or more parallel execution failure
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).ParallelE
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2286
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Parallel
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2167
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:155
		  | github.com/cockroachdb/cockroach/pkg/roachprod.Start
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:662
		  | main.(*clusterImpl).StartE
		  | 	main/pkg/cmd/roachtest/cluster.go:1777
		  | main.(*clusterImpl).Start
		  | 	main/pkg/cmd/roachtest/cluster.go:1822
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:129
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:262
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:906
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (2) one or more parallel execution failure
		Error types: (1) *withstack.withStack (2) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-09-10T06:04:06Z

roachtest.autoupgrade failed with artifacts on master @ 389661e823c19f318fa07ec2278336262531692d:

test artifacts and logs in: /artifacts/autoupgrade/run_1
	cluster.go:1860,autoupgrade.go:129,autoupgrade.go:262,test_runner.go:917: one or more parallel execution failure
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).ParallelE
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2286
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Parallel
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2167
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:155
		  | github.com/cockroachdb/cockroach/pkg/roachprod.Start
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:662
		  | main.(*clusterImpl).StartE
		  | 	main/pkg/cmd/roachtest/cluster.go:1814
		  | main.(*clusterImpl).Start
		  | 	main/pkg/cmd/roachtest/cluster.go:1859
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:129
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:262
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:917
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (2) one or more parallel execution failure
		Error types: (1) *withstack.withStack (2) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-09-11T06:08:37Z

roachtest.autoupgrade failed with artifacts on master @ bc2e47da0523b347c28cf024707e80cd35d6c98a:

test artifacts and logs in: /artifacts/autoupgrade/run_1
	cluster.go:1860,autoupgrade.go:129,autoupgrade.go:262,test_runner.go:917: one or more parallel execution failure
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).ParallelE
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2286
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Parallel
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2167
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:155
		  | github.com/cockroachdb/cockroach/pkg/roachprod.Start
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:662
		  | main.(*clusterImpl).StartE
		  | 	main/pkg/cmd/roachtest/cluster.go:1814
		  | main.(*clusterImpl).Start
		  | 	main/pkg/cmd/roachtest/cluster.go:1859
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:129
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:262
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:917
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (2) one or more parallel execution failure
		Error types: (1) *withstack.withStack (2) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-09-12T06:13:53Z

roachtest.autoupgrade failed with artifacts on master @ 773568fbda06ba9be9fb1bc34a331f21c8891ffa:

test artifacts and logs in: /artifacts/autoupgrade/run_1
	cluster.go:1860,autoupgrade.go:129,autoupgrade.go:262,test_runner.go:917: one or more parallel execution failure
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).ParallelE
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2286
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Parallel
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2167
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:155
		  | github.com/cockroachdb/cockroach/pkg/roachprod.Start
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:662
		  | main.(*clusterImpl).StartE
		  | 	main/pkg/cmd/roachtest/cluster.go:1814
		  | main.(*clusterImpl).Start
		  | 	main/pkg/cmd/roachtest/cluster.go:1859
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:129
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:262
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:917
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (2) one or more parallel execution failure
		Error types: (1) *withstack.withStack (2) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

nvanbenschoten · 2022-09-12T14:41:43Z

This test has started reliably failing with nodes that crash with the log message:

E220909 06:03:14.180386 1 1@cli/clierror/check.go:35 â‹® [-] 19  â€¹ERRORâ€º: cockroach server exited with error: store â€¹node1,node1store1,store1=/mnt/data1/cockroachâ€º, last used with cockroach version v22.1, is too old for running version v1000022.1-70 (which requires data from v1000022.1 or later)

erikgrinaker · 2022-09-12T15:23:33Z

@dt would probably have thoughts here.

cockroach-teamcity · 2022-09-13T06:02:59Z

roachtest.autoupgrade failed with artifacts on master @ 2cfa6b1779ead01508c04e228bf72b4e0e96d98c:

test artifacts and logs in: /artifacts/autoupgrade/run_1
	cluster.go:1860,autoupgrade.go:129,autoupgrade.go:262,test_runner.go:917: one or more parallel execution failure
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).ParallelE
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2286
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Parallel
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2167
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:155
		  | github.com/cockroachdb/cockroach/pkg/roachprod.Start
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:662
		  | main.(*clusterImpl).StartE
		  | 	main/pkg/cmd/roachtest/cluster.go:1814
		  | main.(*clusterImpl).Start
		  | 	main/pkg/cmd/roachtest/cluster.go:1859
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:129
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:262
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:917
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (2) one or more parallel execution failure
		Error types: (1) *withstack.withStack (2) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-09-14T06:03:46Z

roachtest.autoupgrade failed with artifacts on master @ 95677eb5f8d006629b16024fb7d87d55344c1470:

test artifacts and logs in: /artifacts/autoupgrade/run_1
	cluster.go:1860,autoupgrade.go:129,autoupgrade.go:262,test_runner.go:917: one or more parallel execution failure
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).ParallelE
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2286
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Parallel
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2167
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:155
		  | github.com/cockroachdb/cockroach/pkg/roachprod.Start
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:662
		  | main.(*clusterImpl).StartE
		  | 	main/pkg/cmd/roachtest/cluster.go:1814
		  | main.(*clusterImpl).Start
		  | 	main/pkg/cmd/roachtest/cluster.go:1859
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:129
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:262
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:917
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (2) one or more parallel execution failure
		Error types: (1) *withstack.withStack (2) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-09-15T06:04:03Z

roachtest.autoupgrade failed with artifacts on master @ 1f24e52aea4195ffe26b97160079bb29f264338a:

test artifacts and logs in: /artifacts/autoupgrade/run_1
	cluster.go:1860,autoupgrade.go:129,autoupgrade.go:262,test_runner.go:917: one or more parallel execution failure
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).ParallelE
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2286
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Parallel
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2167
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:155
		  | github.com/cockroachdb/cockroach/pkg/roachprod.Start
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:662
		  | main.(*clusterImpl).StartE
		  | 	main/pkg/cmd/roachtest/cluster.go:1814
		  | main.(*clusterImpl).Start
		  | 	main/pkg/cmd/roachtest/cluster.go:1859
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:129
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:262
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:917
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (2) one or more parallel execution failure
		Error types: (1) *withstack.withStack (2) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-09-16T06:07:53Z

roachtest.autoupgrade failed with artifacts on master @ 9a05046ce19e7678340e82c70d61e928be95bc72:

test artifacts and logs in: /artifacts/autoupgrade/run_1
	cluster.go:1860,autoupgrade.go:129,autoupgrade.go:262,test_runner.go:917: one or more parallel execution failure
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).ParallelE
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2286
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Parallel
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2167
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:155
		  | github.com/cockroachdb/cockroach/pkg/roachprod.Start
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:662
		  | main.(*clusterImpl).StartE
		  | 	main/pkg/cmd/roachtest/cluster.go:1814
		  | main.(*clusterImpl).Start
		  | 	main/pkg/cmd/roachtest/cluster.go:1859
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:129
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:262
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:917
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (2) one or more parallel execution failure
		Error types: (1) *withstack.withStack (2) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-09-17T06:09:21Z

roachtest.autoupgrade failed with artifacts on master @ bd97ad5b8c9f537a89492a051574d867469bef33:

test artifacts and logs in: /artifacts/autoupgrade/run_1
	cluster.go:1860,autoupgrade.go:129,autoupgrade.go:262,test_runner.go:917: one or more parallel execution failure
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).ParallelE
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2286
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Parallel
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2167
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:155
		  | github.com/cockroachdb/cockroach/pkg/roachprod.Start
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:662
		  | main.(*clusterImpl).StartE
		  | 	main/pkg/cmd/roachtest/cluster.go:1814
		  | main.(*clusterImpl).Start
		  | 	main/pkg/cmd/roachtest/cluster.go:1859
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:129
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:262
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:917
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (2) one or more parallel execution failure
		Error types: (1) *withstack.withStack (2) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-09-18T06:07:58Z

roachtest.autoupgrade failed with artifacts on master @ 3fa1a1600898d7b78b9e39d07132a387a2f9a1b6:

test artifacts and logs in: /artifacts/autoupgrade/run_1
	cluster.go:1860,autoupgrade.go:129,autoupgrade.go:262,test_runner.go:917: one or more parallel execution failure
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).ParallelE
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2286
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Parallel
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2167
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:155
		  | github.com/cockroachdb/cockroach/pkg/roachprod.Start
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:662
		  | main.(*clusterImpl).StartE
		  | 	main/pkg/cmd/roachtest/cluster.go:1814
		  | main.(*clusterImpl).Start
		  | 	main/pkg/cmd/roachtest/cluster.go:1859
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:129
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:262
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:917
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (2) one or more parallel execution failure
		Error types: (1) *withstack.withStack (2) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-09-19T06:08:40Z

roachtest.autoupgrade failed with artifacts on master @ 87ed064dc23eab6948ee8a07e8507f150bda0e44:

test artifacts and logs in: /artifacts/autoupgrade/run_1
	cluster.go:1860,autoupgrade.go:129,autoupgrade.go:262,test_runner.go:917: one or more parallel execution failure
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).ParallelE
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2286
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Parallel
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2167
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:155
		  | github.com/cockroachdb/cockroach/pkg/roachprod.Start
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:662
		  | main.(*clusterImpl).StartE
		  | 	main/pkg/cmd/roachtest/cluster.go:1814
		  | main.(*clusterImpl).Start
		  | 	main/pkg/cmd/roachtest/cluster.go:1859
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:129
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:262
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:917
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (2) one or more parallel execution failure
		Error types: (1) *withstack.withStack (2) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

erikgrinaker · 2022-09-19T08:13:33Z

Reassigning this to @dt for the most recent failure mode:

E220909 06:03:14.180386 1 1@cli/clierror/check.go:35 â‹® [-] 19  â€¹ERRORâ€º: cockroach server exited with error: store â€¹node1,node1store1,store1=/mnt/data1/cockroachâ€º, last used with cockroach version v22.1, is too old for running version v1000022.1-70 (which requires data from v1000022.1 or later)

irfansharif · 2022-09-19T13:47:56Z

Perhaps a result of #86345?

erikgrinaker · 2022-09-19T13:57:09Z

Yeah, seems related to that work.

dt · 2022-09-19T14:15:00Z

I think @srosenberg has a blanket fix for these in #88005

dt · 2022-09-19T14:24:42Z

also the version thing isn't a ga or release blocker since it only affects master.

erikgrinaker · 2022-09-19T14:53:56Z

I think the GA blocker here was for the original failure, before the version upgrades started failing. Re-labeling, but @nvanbenschoten can feel free to remove otherwise.

nvanbenschoten · 2022-09-26T14:51:50Z

Closing, as we have not seen the original failure mode in over a month and Aayush was unable to reproduce in #86520 (comment).

cockroach-teamcity added this to the 22.2 milestone Aug 21, 2022

blathers-crl bot added the T-kv KV Team label Aug 21, 2022

aayushshah15 self-assigned this Aug 29, 2022

nvanbenschoten added GA-blocker and removed release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Sep 6, 2022

erikgrinaker assigned dt and unassigned aayushshah15 Sep 19, 2022

dt removed the GA-blocker label Sep 19, 2022

erikgrinaker added the GA-blocker label Sep 19, 2022

nvanbenschoten closed this as completed Sep 26, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

roachtest: autoupgrade failed #86520

roachtest: autoupgrade failed #86520

cockroach-teamcity commented Aug 21, 2022 •

edited by exalate-issue-sync bot

Loading

cockroach-teamcity commented Aug 23, 2022

cockroach-teamcity commented Aug 24, 2022

aayushshah15 commented Sep 6, 2022

cockroach-teamcity commented Sep 9, 2022

cockroach-teamcity commented Sep 10, 2022

cockroach-teamcity commented Sep 11, 2022

cockroach-teamcity commented Sep 12, 2022

nvanbenschoten commented Sep 12, 2022

erikgrinaker commented Sep 12, 2022

cockroach-teamcity commented Sep 13, 2022

cockroach-teamcity commented Sep 14, 2022

cockroach-teamcity commented Sep 15, 2022

cockroach-teamcity commented Sep 16, 2022

cockroach-teamcity commented Sep 17, 2022

cockroach-teamcity commented Sep 18, 2022

cockroach-teamcity commented Sep 19, 2022

erikgrinaker commented Sep 19, 2022

irfansharif commented Sep 19, 2022

erikgrinaker commented Sep 19, 2022

dt commented Sep 19, 2022

dt commented Sep 19, 2022

erikgrinaker commented Sep 19, 2022 •

edited

Loading

nvanbenschoten commented Sep 26, 2022

roachtest: autoupgrade failed #86520

roachtest: autoupgrade failed #86520

Comments

cockroach-teamcity commented Aug 21, 2022 • edited by exalate-issue-sync bot Loading

cockroach-teamcity commented Aug 23, 2022

cockroach-teamcity commented Aug 24, 2022

aayushshah15 commented Sep 6, 2022

cockroach-teamcity commented Sep 9, 2022

cockroach-teamcity commented Sep 10, 2022

cockroach-teamcity commented Sep 11, 2022

cockroach-teamcity commented Sep 12, 2022

nvanbenschoten commented Sep 12, 2022

erikgrinaker commented Sep 12, 2022

cockroach-teamcity commented Sep 13, 2022

cockroach-teamcity commented Sep 14, 2022

cockroach-teamcity commented Sep 15, 2022

cockroach-teamcity commented Sep 16, 2022

cockroach-teamcity commented Sep 17, 2022

cockroach-teamcity commented Sep 18, 2022

cockroach-teamcity commented Sep 19, 2022

erikgrinaker commented Sep 19, 2022

irfansharif commented Sep 19, 2022

erikgrinaker commented Sep 19, 2022

dt commented Sep 19, 2022

dt commented Sep 19, 2022

erikgrinaker commented Sep 19, 2022 • edited Loading

nvanbenschoten commented Sep 26, 2022

cockroach-teamcity commented Aug 21, 2022 •

edited by exalate-issue-sync bot

Loading

erikgrinaker commented Sep 19, 2022 •

edited

Loading