roachtest: multitenant-upgrade failed #81517

cockroach-teamcity · 2022-05-19T11:43:40Z

roachtest.multitenant-upgrade failed with artifacts on master @ ce66acdbba801de88f1dd645eaedeb3834f23dbd:

		Wraps: (2) cluster.RunE
		Wraps: (3) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

	cluster.go:1936,multitenant_utils.go:92,panic.go:1038,test_impl.go:254,cluster.go:1936,multitenant_utils.go:92,panic.go:1038,test_impl.go:254,cluster.go:1936,multitenant_utils.go:92,panic.go:1038,test_impl.go:260,sql_runner.go:148,sql_runner.go:177,multitenant_upgrade.go:314,multitenant_upgrade.go:177,multitenant_upgrade.go:35,test_runner.go:876: cluster.RunE: context canceled
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1953
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1934
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:92
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:1038
		  | main.(*testImpl).Fatal
		  | 	main/pkg/cmd/roachtest/test_impl.go:254
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1936
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:92
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:1038
		  | main.(*testImpl).Fatal
		  | 	main/pkg/cmd/roachtest/test_impl.go:254
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1936
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:92
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:1038
		  | main.(*testImpl).Fatalf
		  | 	main/pkg/cmd/roachtest/test_impl.go:260
		  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).Query
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:148
		  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).QueryStr
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:177
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.verifySQL
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:314
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:177
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:35
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:876
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) cluster.RunE
		Wraps: (3) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/kv-triage _{This test on roachdash | Improve this report!

Jira issue: CRDB-15107}

The text was updated successfully, but these errors were encountered:

cockroach-teamcity · 2022-05-21T11:18:59Z

roachtest.multitenant-upgrade failed with artifacts on master @ 7dd1c51f6b5802e32bafd82e46747f349836592f:

		Wraps: (2) cluster.RunE
		Wraps: (3) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

	cluster.go:1913,multitenant_utils.go:92,panic.go:1038,test_impl.go:254,cluster.go:1913,multitenant_utils.go:92,panic.go:1038,test_impl.go:254,cluster.go:1913,multitenant_utils.go:92,panic.go:1038,test_impl.go:260,sql_runner.go:148,sql_runner.go:177,multitenant_upgrade.go:314,multitenant_upgrade.go:177,multitenant_upgrade.go:35,test_runner.go:884: cluster.RunE: context canceled
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1930
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1911
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:92
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:1038
		  | main.(*testImpl).Fatal
		  | 	main/pkg/cmd/roachtest/test_impl.go:254
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1913
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:92
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:1038
		  | main.(*testImpl).Fatal
		  | 	main/pkg/cmd/roachtest/test_impl.go:254
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1913
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:92
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:1038
		  | main.(*testImpl).Fatalf
		  | 	main/pkg/cmd/roachtest/test_impl.go:260
		  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).Query
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:148
		  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).QueryStr
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:177
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.verifySQL
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:314
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:177
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:35
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:884
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) cluster.RunE
		Wraps: (3) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

lidorcarmel · 2022-05-24T03:02:06Z

right before the stack trace:

11:43:18 multitenant_upgrade.go:119: test status: checking the pre-upgrade sql server still works after the KV binary upgrade
11:43:18 multitenant_upgrade.go:125: test status: creating a new tenant 12
11:43:18 multitenant_upgrade.go:131: test status: starting tenant 12 server with older binary
11:43:22 multitenant_utils.go:173: sql server for tenant 12 running at postgres://[email protected]:20012?sslcert=%2Ftmp%2Froachtest-certs4016127605%2Fcerts%2Fclient.root.crt&sslkey=%2Ftmp%2Froachtest-certs4016127605%2Fcerts%2Fclient.root.key&sslmode=verify-full&sslrootcert=%2Ftmp%2Froachtest-certs4016127605%2Fcerts%2Fca.crt
11:43:22 multitenant_upgrade.go:136: test status: verifying that the tenant 12 server works and is at the earlier version
11:43:22 multitenant_upgrade.go:147: test status: creating a new tenant 13
11:43:22 multitenant_upgrade.go:153: test status: starting tenant 13 server with new binary
11:43:26 multitenant_utils.go:173: sql server for tenant 13 running at postgres://[email protected]:20013?sslcert=%2Ftmp%2Froachtest-certs4016127605%2Fcerts%2Fclient.root.crt&sslkey=%2Ftmp%2Froachtest-certs4016127605%2Fcerts%2Fclient.root.key&sslmode=verify-full&sslrootcert=%2Ftmp%2Froachtest-certs4016127605%2Fcerts%2Fca.crt
11:43:26 multitenant_upgrade.go:158: test status: verifying that the tenant 13 server works and is at the earlier version
11:43:26 multitenant_upgrade.go:169: test status: stopping the tenant 11 server ahead of upgrading
11:43:27 multitenant_utils.go:94: mt cluster exited: output in run_114310.196065487_n2_v2127cockroach_mt: v21.2.7/cockroach mt start-sql --certs-dir certs --tenant-id=11 --http-addr 0.0.0.0:8011 --kv-addrs 10.142.0.150:26257 --sql-addr 10.142.0.141:20011 --log="file-defaults: {dir: 'logs/mt-11', exit-on-error: false}" --store=cockroach-data-mt-11 returned: COMMAND_PROBLEM: exit status 1
11:43:27 multitenant_upgrade.go:172: test status: starting the tenant 11 server with the current binary
11:43:29 multitenant_utils.go:173: sql server for tenant 11 running at postgres://[email protected]:20011?sslcert=%2Ftmp%2Froachtest-certs4016127605%2Fcerts%2Fclient.root.crt&sslkey=%2Ftmp%2Froachtest-certs4016127605%2Fcerts%2Fclient.root.key&sslmode=verify-full&sslrootcert=%2Ftmp%2Froachtest-certs4016127605%2Fcerts%2Fca.crt
11:43:29 multitenant_upgrade.go:175: test status: verify tenant 11 server works with the new binary
11:43:29 test_impl.go:318: test failure: 	sql_runner.go:148,sql_runner.go:177,multitenant_upgrade.go:314,multitenant_upgrade.go:177,multitenant_upgrade.go:35,test_runner.go:876: error executing 'SELECT * FROM foo LIMIT 1': pq: failed to connect to n1 at 10.142.0.141:20011: initial connection heartbeat failed: rpc error: code = Unknown desc = client requested node ID 1 doesn't match server node ID 2
11:43:29 test_impl.go:318: [not the first failure] test failure: 
	cluster.go:1936,multitenant_utils.go:92,panic.go:1038,test_impl.go:260,sql_runner.go:148,sql_runner.go:177,multitenant_upgrade.go:314,multitenant_upgrade.go:177,multitenant_upgrade.go:35,test_runner.go:876: cluster.RunE: context canceled

tbg · 2022-05-24T09:51:16Z

It's from here:

cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go

Lines 176 to 182 in ed4c6b4

    
           { 
        
           	verifySQL(t, tenant11.pgURL, 
        
           		mkStmt(`SELECT * FROM foo LIMIT 1`). 
        
           			withResults([][]string{{"1", "bar"}}), 
        
           		mkStmt("SHOW CLUSTER SETTING version"). 
        
           			withResults([][]string{{initialVersion}})) 
        
           }

so in serving this query, someone (the tenant pod for t11 probably) was trying to dial n1 but ended up reaching n2. Seems unlikely since we're not shuffling nodes around their addresses here.

I wonder if there's some confusion going on regarding SQL instance IDs vs NodeIDs.

cockroach-teamcity · 2022-05-24T11:43:12Z

roachtest.multitenant-upgrade failed with artifacts on master @ 48e48db89eb3cd6bc38f3631364c516181811280:

		Wraps: (2) cluster.RunE
		Wraps: (3) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

	cluster.go:1913,multitenant_utils.go:92,panic.go:1038,test_impl.go:254,cluster.go:1913,multitenant_utils.go:92,panic.go:1038,test_impl.go:254,cluster.go:1913,multitenant_utils.go:92,panic.go:1038,test_impl.go:260,sql_runner.go:148,sql_runner.go:177,multitenant_upgrade.go:314,multitenant_upgrade.go:177,multitenant_upgrade.go:35,test_runner.go:884: cluster.RunE: context canceled
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1930
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1911
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:92
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:1038
		  | main.(*testImpl).Fatal
		  | 	main/pkg/cmd/roachtest/test_impl.go:254
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1913
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:92
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:1038
		  | main.(*testImpl).Fatal
		  | 	main/pkg/cmd/roachtest/test_impl.go:254
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1913
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:92
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:1038
		  | main.(*testImpl).Fatalf
		  | 	main/pkg/cmd/roachtest/test_impl.go:260
		  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).Query
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:148
		  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).QueryStr
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:177
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.verifySQL
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:314
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:177
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:35
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:884
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) cluster.RunE
		Wraps: (3) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-05-25T11:46:41Z

roachtest.multitenant-upgrade failed with artifacts on master @ 12198a51408e7333cd4f96b221e6734239479765:

		Wraps: (2) cluster.RunE
		Wraps: (3) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

	cluster.go:1913,multitenant_utils.go:92,panic.go:1038,test_impl.go:254,cluster.go:1913,multitenant_utils.go:92,panic.go:1038,test_impl.go:254,cluster.go:1913,multitenant_utils.go:92,panic.go:1038,test_impl.go:260,sql_runner.go:148,sql_runner.go:177,multitenant_upgrade.go:314,multitenant_upgrade.go:177,multitenant_upgrade.go:35,test_runner.go:884: cluster.RunE: context canceled
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1930
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1911
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:92
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:1038
		  | main.(*testImpl).Fatal
		  | 	main/pkg/cmd/roachtest/test_impl.go:254
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1913
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:92
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:1038
		  | main.(*testImpl).Fatal
		  | 	main/pkg/cmd/roachtest/test_impl.go:254
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1913
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:92
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:1038
		  | main.(*testImpl).Fatalf
		  | 	main/pkg/cmd/roachtest/test_impl.go:260
		  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).Query
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:148
		  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).QueryStr
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:177
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.verifySQL
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:314
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:177
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:35
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:884
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) cluster.RunE
		Wraps: (3) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-05-26T11:49:48Z

roachtest.multitenant-upgrade failed with artifacts on master @ 7a17498a9679853612cb88d82a4a3952d1015f94:

		Wraps: (2) cluster.RunE
		Wraps: (3) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

	cluster.go:1913,multitenant_utils.go:92,panic.go:1038,test_impl.go:254,cluster.go:1913,multitenant_utils.go:92,panic.go:1038,test_impl.go:254,cluster.go:1913,multitenant_utils.go:92,panic.go:1038,test_impl.go:260,sql_runner.go:148,sql_runner.go:177,multitenant_upgrade.go:314,multitenant_upgrade.go:177,multitenant_upgrade.go:35,test_runner.go:884: cluster.RunE: context canceled
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1930
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1911
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:92
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:1038
		  | main.(*testImpl).Fatal
		  | 	main/pkg/cmd/roachtest/test_impl.go:254
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1913
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:92
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:1038
		  | main.(*testImpl).Fatal
		  | 	main/pkg/cmd/roachtest/test_impl.go:254
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1913
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:92
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:1038
		  | main.(*testImpl).Fatalf
		  | 	main/pkg/cmd/roachtest/test_impl.go:260
		  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).Query
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:148
		  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).QueryStr
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:177
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.verifySQL
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:314
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:177
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:35
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:884
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) cluster.RunE
		Wraps: (3) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-05-27T11:41:25Z

roachtest.multitenant-upgrade failed with artifacts on master @ d9fe5f67c75b1b59fc297bf4509a139c640b835b:

		Wraps: (2) cluster.RunE
		Wraps: (3) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

	cluster.go:1913,multitenant_utils.go:92,panic.go:1038,test_impl.go:254,cluster.go:1913,multitenant_utils.go:92,panic.go:1038,test_impl.go:254,cluster.go:1913,multitenant_utils.go:92,panic.go:1038,test_impl.go:260,sql_runner.go:148,sql_runner.go:177,multitenant_upgrade.go:314,multitenant_upgrade.go:177,multitenant_upgrade.go:35,test_runner.go:884: cluster.RunE: context canceled
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1930
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1911
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:92
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:1038
		  | main.(*testImpl).Fatal
		  | 	main/pkg/cmd/roachtest/test_impl.go:254
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1913
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:92
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:1038
		  | main.(*testImpl).Fatal
		  | 	main/pkg/cmd/roachtest/test_impl.go:254
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1913
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:92
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:1038
		  | main.(*testImpl).Fatalf
		  | 	main/pkg/cmd/roachtest/test_impl.go:260
		  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).Query
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:148
		  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).QueryStr
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:177
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.verifySQL
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:314
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:177
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:35
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:884
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) cluster.RunE
		Wraps: (3) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-05-28T11:58:27Z

roachtest.multitenant-upgrade failed with artifacts on master @ 1e2cc61b58dc14386bb68dca59814874648931c2:

		Wraps: (2) cluster.RunE
		Wraps: (3) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

	cluster.go:1913,multitenant_utils.go:92,panic.go:1038,test_impl.go:254,cluster.go:1913,multitenant_utils.go:92,panic.go:1038,test_impl.go:254,cluster.go:1913,multitenant_utils.go:92,panic.go:1038,test_impl.go:260,sql_runner.go:148,sql_runner.go:177,multitenant_upgrade.go:314,multitenant_upgrade.go:177,multitenant_upgrade.go:35,test_runner.go:884: cluster.RunE: context canceled
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1930
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1911
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:92
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:1038
		  | main.(*testImpl).Fatal
		  | 	main/pkg/cmd/roachtest/test_impl.go:254
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1913
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:92
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:1038
		  | main.(*testImpl).Fatal
		  | 	main/pkg/cmd/roachtest/test_impl.go:254
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1913
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:92
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:1038
		  | main.(*testImpl).Fatalf
		  | 	main/pkg/cmd/roachtest/test_impl.go:260
		  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).Query
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:148
		  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).QueryStr
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:177
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.verifySQL
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:314
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:177
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:35
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:884
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) cluster.RunE
		Wraps: (3) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-05-29T11:27:51Z

roachtest.multitenant-upgrade failed with artifacts on master @ e6815947a050e32f21e983aa30dc74ab2a247af3:

		Wraps: (2) cluster.RunE
		Wraps: (3) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

	cluster.go:1913,multitenant_utils.go:92,panic.go:1038,test_impl.go:254,cluster.go:1913,multitenant_utils.go:92,panic.go:1038,test_impl.go:254,cluster.go:1913,multitenant_utils.go:92,panic.go:1038,test_impl.go:260,sql_runner.go:148,sql_runner.go:177,multitenant_upgrade.go:314,multitenant_upgrade.go:177,multitenant_upgrade.go:35,test_runner.go:884: cluster.RunE: context canceled
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1930
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1911
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:92
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:1038
		  | main.(*testImpl).Fatal
		  | 	main/pkg/cmd/roachtest/test_impl.go:254
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1913
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:92
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:1038
		  | main.(*testImpl).Fatal
		  | 	main/pkg/cmd/roachtest/test_impl.go:254
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1913
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:92
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:1038
		  | main.(*testImpl).Fatalf
		  | 	main/pkg/cmd/roachtest/test_impl.go:260
		  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).Query
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:148
		  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).QueryStr
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:177
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.verifySQL
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:314
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:177
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:35
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:884
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) cluster.RunE
		Wraps: (3) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-05-30T11:49:59Z

roachtest.multitenant-upgrade failed with artifacts on master @ e6815947a050e32f21e983aa30dc74ab2a247af3:

		Wraps: (2) cluster.RunE
		Wraps: (3) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

	cluster.go:1913,multitenant_utils.go:92,panic.go:1038,test_impl.go:254,cluster.go:1913,multitenant_utils.go:92,panic.go:1038,test_impl.go:254,cluster.go:1913,multitenant_utils.go:92,panic.go:1038,test_impl.go:260,sql_runner.go:148,sql_runner.go:177,multitenant_upgrade.go:314,multitenant_upgrade.go:177,multitenant_upgrade.go:35,test_runner.go:884: cluster.RunE: context canceled
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1930
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1911
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:92
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:1038
		  | main.(*testImpl).Fatal
		  | 	main/pkg/cmd/roachtest/test_impl.go:254
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1913
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:92
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:1038
		  | main.(*testImpl).Fatal
		  | 	main/pkg/cmd/roachtest/test_impl.go:254
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1913
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:92
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:1038
		  | main.(*testImpl).Fatalf
		  | 	main/pkg/cmd/roachtest/test_impl.go:260
		  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).Query
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:148
		  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).QueryStr
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:177
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.verifySQL
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:314
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:177
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:35
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:884
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) cluster.RunE
		Wraps: (3) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-05-31T12:06:34Z

roachtest.multitenant-upgrade failed with artifacts on master @ 662cc5c3070e6d64d155a9cc9f33253ee5d99ee9:

		Wraps: (2) cluster.RunE
		Wraps: (3) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

	cluster.go:1913,multitenant_utils.go:92,panic.go:1038,test_impl.go:254,cluster.go:1913,multitenant_utils.go:92,panic.go:1038,test_impl.go:254,cluster.go:1913,multitenant_utils.go:92,panic.go:1038,test_impl.go:260,sql_runner.go:148,sql_runner.go:177,multitenant_upgrade.go:314,multitenant_upgrade.go:177,multitenant_upgrade.go:35,test_runner.go:884: cluster.RunE: context canceled
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1930
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1911
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:92
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:1038
		  | main.(*testImpl).Fatal
		  | 	main/pkg/cmd/roachtest/test_impl.go:254
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1913
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:92
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:1038
		  | main.(*testImpl).Fatal
		  | 	main/pkg/cmd/roachtest/test_impl.go:254
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1913
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:92
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:1038
		  | main.(*testImpl).Fatalf
		  | 	main/pkg/cmd/roachtest/test_impl.go:260
		  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).Query
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:148
		  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).QueryStr
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:177
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.verifySQL
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:314
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:177
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:35
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:884
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) cluster.RunE
		Wraps: (3) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: multitenant-upgrade failed #82122 roachtest: multitenant-upgrade failed [C-test-failure O-roachtest O-robot T-kv branch-release-22.1 release-blocker]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-06-01T11:44:30Z

roachtest.multitenant-upgrade failed with artifacts on master @ 1cea73c8a18623949b81705eb5f75179e6cd8d86:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /artifacts/multitenant-upgrade/run_1
	test_runner.go:879,panic.go:1038,version.go:125,multitenant_utils.go:72,multitenant_upgrade.go:88,multitenant_upgrade.go:35,test_runner.go:884: test panicked: invalid version string '22.1.0'

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: multitenant-upgrade failed #82122 roachtest: multitenant-upgrade failed [C-test-failure O-roachtest O-robot T-kv branch-release-22.1 release-blocker]

_{This test on roachdash | Improve this report!}

tbg · 2022-06-02T11:24:31Z

@mwang1026 this should be given some attention, it's failing like clockwork and multitenant upgrades are important.

cockroach-teamcity · 2022-06-02T12:40:47Z

roachtest.multitenant-upgrade failed with artifacts on master @ 5a54758ce89e866b4fe28c0df74bd610973c6918:

		Wraps: (2) cluster.RunE
		Wraps: (3) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

	cluster.go:1914,multitenant_utils.go:111,panic.go:1038,test_impl.go:254,cluster.go:1914,multitenant_utils.go:111,panic.go:1038,test_impl.go:254,cluster.go:1914,multitenant_utils.go:111,panic.go:1038,test_impl.go:260,sql_runner.go:148,sql_runner.go:177,multitenant_upgrade.go:314,multitenant_upgrade.go:177,multitenant_upgrade.go:35,test_runner.go:884: cluster.RunE: context canceled
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1931
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1912
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:111
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:1038
		  | main.(*testImpl).Fatal
		  | 	main/pkg/cmd/roachtest/test_impl.go:254
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1914
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:111
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:1038
		  | main.(*testImpl).Fatal
		  | 	main/pkg/cmd/roachtest/test_impl.go:254
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1914
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:111
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:1038
		  | main.(*testImpl).Fatalf
		  | 	main/pkg/cmd/roachtest/test_impl.go:260
		  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).Query
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:148
		  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).QueryStr
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:177
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.verifySQL
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:314
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:177
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:35
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:884
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) cluster.RunE
		Wraps: (3) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-06-03T12:03:21Z

roachtest.multitenant-upgrade failed with artifacts on master @ 2181204e9c7ac6b316573073b6b8010f43920f8b:

		Wraps: (2) cluster.RunE
		Wraps: (3) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

	cluster.go:1914,multitenant_utils.go:111,panic.go:1038,test_impl.go:254,cluster.go:1914,multitenant_utils.go:111,panic.go:1038,test_impl.go:254,cluster.go:1914,multitenant_utils.go:111,panic.go:1038,test_impl.go:260,sql_runner.go:148,sql_runner.go:177,multitenant_upgrade.go:314,multitenant_upgrade.go:177,multitenant_upgrade.go:35,test_runner.go:884: cluster.RunE: context canceled
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1931
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1912
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:111
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:1038
		  | main.(*testImpl).Fatal
		  | 	main/pkg/cmd/roachtest/test_impl.go:254
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1914
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:111
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:1038
		  | main.(*testImpl).Fatal
		  | 	main/pkg/cmd/roachtest/test_impl.go:254
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1914
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:111
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:1038
		  | main.(*testImpl).Fatalf
		  | 	main/pkg/cmd/roachtest/test_impl.go:260
		  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).Query
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:148
		  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).QueryStr
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:177
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.verifySQL
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:314
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:177
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:35
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:884
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) cluster.RunE
		Wraps: (3) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-06-04T11:51:58Z

roachtest.multitenant-upgrade failed with artifacts on master @ 90b5db50e1e1cdb0315d8b094081d261e6dcb336:

		Wraps: (2) cluster.RunE
		Wraps: (3) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

	cluster.go:1914,multitenant_utils.go:111,panic.go:1038,test_impl.go:254,cluster.go:1914,multitenant_utils.go:111,panic.go:1038,test_impl.go:254,cluster.go:1914,multitenant_utils.go:111,panic.go:1038,test_impl.go:260,sql_runner.go:148,sql_runner.go:177,multitenant_upgrade.go:314,multitenant_upgrade.go:177,multitenant_upgrade.go:35,test_runner.go:884: cluster.RunE: context canceled
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1931
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1912
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:111
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:1038
		  | main.(*testImpl).Fatal
		  | 	main/pkg/cmd/roachtest/test_impl.go:254
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1914
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:111
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:1038
		  | main.(*testImpl).Fatal
		  | 	main/pkg/cmd/roachtest/test_impl.go:254
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1914
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:111
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:1038
		  | main.(*testImpl).Fatalf
		  | 	main/pkg/cmd/roachtest/test_impl.go:260
		  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).Query
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:148
		  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).QueryStr
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:177
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.verifySQL
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:314
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:177
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:35
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:884
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) cluster.RunE
		Wraps: (3) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-06-05T12:50:20Z

roachtest.multitenant-upgrade failed with artifacts on master @ 0fdac09ca1119b494661e1e1f64ea291d8649782:

		Wraps: (2) cluster.RunE
		Wraps: (3) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

	cluster.go:1914,multitenant_utils.go:111,panic.go:1038,test_impl.go:254,cluster.go:1914,multitenant_utils.go:111,panic.go:1038,test_impl.go:254,cluster.go:1914,multitenant_utils.go:111,panic.go:1038,test_impl.go:260,sql_runner.go:148,sql_runner.go:177,multitenant_upgrade.go:314,multitenant_upgrade.go:177,multitenant_upgrade.go:35,test_runner.go:884: cluster.RunE: context canceled
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1931
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1912
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:111
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:1038
		  | main.(*testImpl).Fatal
		  | 	main/pkg/cmd/roachtest/test_impl.go:254
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1914
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:111
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:1038
		  | main.(*testImpl).Fatal
		  | 	main/pkg/cmd/roachtest/test_impl.go:254
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1914
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:111
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:1038
		  | main.(*testImpl).Fatalf
		  | 	main/pkg/cmd/roachtest/test_impl.go:260
		  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).Query
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:148
		  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).QueryStr
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:177
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.verifySQL
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:314
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:177
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:35
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:884
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) cluster.RunE
		Wraps: (3) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-06-06T12:13:01Z

roachtest.multitenant-upgrade failed with artifacts on master @ 2c08debd6d8c89019f26a3e58cf3b5d3d97b6495:

		Wraps: (2) cluster.RunE
		Wraps: (3) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

	cluster.go:1914,multitenant_utils.go:111,panic.go:1038,test_impl.go:254,cluster.go:1914,multitenant_utils.go:111,panic.go:1038,test_impl.go:254,cluster.go:1914,multitenant_utils.go:111,panic.go:1038,test_impl.go:260,sql_runner.go:148,sql_runner.go:177,multitenant_upgrade.go:314,multitenant_upgrade.go:177,multitenant_upgrade.go:35,test_runner.go:884: cluster.RunE: context canceled
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1931
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1912
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:111
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:1038
		  | main.(*testImpl).Fatal
		  | 	main/pkg/cmd/roachtest/test_impl.go:254
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1914
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:111
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:1038
		  | main.(*testImpl).Fatal
		  | 	main/pkg/cmd/roachtest/test_impl.go:254
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1914
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:111
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:1038
		  | main.(*testImpl).Fatalf
		  | 	main/pkg/cmd/roachtest/test_impl.go:260
		  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).Query
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:148
		  | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).QueryStr
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:177
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.verifySQL
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:314
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:177
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:35
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:884
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) cluster.RunE
		Wraps: (3) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-08-13T13:41:35Z

roachtest.multitenant-upgrade failed with artifacts on master @ 0dd438d3dc0b42543890455945a7a6b42811def1:

		  | SQLSTATE: 42704
		  | Failed running "mt start-sql"
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 2. Command with error:
		  | ``````
		  | v22.1.5/cockroach mt start-sql --certs-dir certs --tenant-id=12 --http-addr 0.0.0.0:8012 --kv-addrs 10.142.0.130:26257 --sql-addr 10.142.0.133:20012 --log="file-defaults: {dir: 'logs/mt-12', exit-on-error: false}" --store=cockroach-data-mt-12
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

	cluster.go:1931,multitenant_utils.go:144,panic.go:838,test_impl.go:254,multitenant_utils.go:210,soon.go:69,retry.go:207,soon.go:75,soon.go:57,soon.go:40,multitenant_utils.go:205,multitenant_upgrade.go:134,multitenant_upgrade.go:35,test_runner.go:896: cluster.RunE: context canceled
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1948
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1929
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:144
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:838
		  | main.(*testImpl).Fatal
		  | 	main/pkg/cmd/roachtest/test_impl.go:254
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).start.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:210
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError.func1
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:69
		  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
		  | 	github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:207
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithin
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:57
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:40
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).start
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:205
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:134
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:35
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:896
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (2) cluster.RunE
		Wraps: (3) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-08-14T13:34:33Z

roachtest.multitenant-upgrade failed with artifacts on master @ d25cb57ccd9bc643ce9058ebd2057cab36b69ad5:

		  | SQLSTATE: 42704
		  | Failed running "mt start-sql"
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 2. Command with error:
		  | ``````
		  | v22.1.5/cockroach mt start-sql --certs-dir certs --tenant-id=12 --http-addr 0.0.0.0:8012 --kv-addrs 10.142.0.131:26257 --sql-addr 10.142.0.136:20012 --log="file-defaults: {dir: 'logs/mt-12', exit-on-error: false}" --store=cockroach-data-mt-12
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

	cluster.go:1931,multitenant_utils.go:144,panic.go:838,test_impl.go:254,multitenant_utils.go:210,soon.go:69,retry.go:207,soon.go:75,soon.go:57,soon.go:40,multitenant_utils.go:205,multitenant_upgrade.go:134,multitenant_upgrade.go:35,test_runner.go:896: cluster.RunE: context canceled
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1948
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1929
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:144
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:838
		  | main.(*testImpl).Fatal
		  | 	main/pkg/cmd/roachtest/test_impl.go:254
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).start.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:210
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError.func1
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:69
		  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
		  | 	github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:207
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithin
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:57
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:40
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).start
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:205
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:134
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:35
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:896
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (2) cluster.RunE
		Wraps: (3) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-08-15T12:52:43Z

roachtest.multitenant-upgrade failed with artifacts on master @ 41db784cb97d2749b162020c2c821979094f87b1:

		  | SQLSTATE: 42704
		  | Failed running "mt start-sql"
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 2. Command with error:
		  | ``````
		  | v22.1.5/cockroach mt start-sql --certs-dir certs --tenant-id=12 --http-addr 0.0.0.0:8012 --kv-addrs 10.142.0.107:26257 --sql-addr 10.142.0.127:20012 --log="file-defaults: {dir: 'logs/mt-12', exit-on-error: false}" --store=cockroach-data-mt-12
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

	cluster.go:1931,multitenant_utils.go:144,panic.go:838,test_impl.go:254,multitenant_utils.go:210,soon.go:69,retry.go:207,soon.go:75,soon.go:57,soon.go:40,multitenant_utils.go:205,multitenant_upgrade.go:134,multitenant_upgrade.go:35,test_runner.go:896: cluster.RunE: context canceled
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1948
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1929
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:144
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:838
		  | main.(*testImpl).Fatal
		  | 	main/pkg/cmd/roachtest/test_impl.go:254
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).start.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:210
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError.func1
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:69
		  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
		  | 	github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:207
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithin
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:57
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:40
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).start
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:205
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:134
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:35
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:896
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (2) cluster.RunE
		Wraps: (3) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-08-16T13:31:56Z

roachtest.multitenant-upgrade failed with artifacts on master @ f4042d47fa8062a612c38d4696eb6bee9cee7c21:

		  | SQLSTATE: 42704
		  | Failed running "mt start-sql"
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 2. Command with error:
		  | ``````
		  | v22.1.5/cockroach mt start-sql --certs-dir certs --tenant-id=12 --http-addr 0.0.0.0:8012 --kv-addrs 10.142.0.153:26257 --sql-addr 10.142.0.111:20012 --log="file-defaults: {dir: 'logs/mt-12', exit-on-error: false}" --store=cockroach-data-mt-12
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

	cluster.go:1931,multitenant_utils.go:144,panic.go:838,test_impl.go:254,multitenant_utils.go:210,soon.go:69,retry.go:207,soon.go:75,soon.go:57,soon.go:40,multitenant_utils.go:205,multitenant_upgrade.go:134,multitenant_upgrade.go:35,test_runner.go:896: cluster.RunE: context canceled
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1948
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1929
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:144
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:838
		  | main.(*testImpl).Fatal
		  | 	main/pkg/cmd/roachtest/test_impl.go:254
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).start.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:210
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError.func1
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:69
		  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
		  | 	github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:207
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithin
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:57
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:40
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).start
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:205
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:134
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:35
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:896
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (2) cluster.RunE
		Wraps: (3) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-08-17T13:10:00Z

roachtest.multitenant-upgrade failed with artifacts on master @ b173a16715e71e94115820374da1eb350b3b459d:

		  | SQLSTATE: 42704
		  | Failed running "mt start-sql"
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 2. Command with error:
		  | ``````
		  | v22.1.5/cockroach mt start-sql --certs-dir certs --tenant-id=12 --http-addr 0.0.0.0:8012 --kv-addrs 10.142.0.174:26257 --sql-addr 10.142.0.186:20012 --log="file-defaults: {dir: 'logs/mt-12', exit-on-error: false}" --store=cockroach-data-mt-12
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

	cluster.go:1931,multitenant_utils.go:144,panic.go:838,test_impl.go:254,multitenant_utils.go:210,soon.go:69,retry.go:207,soon.go:75,soon.go:57,soon.go:40,multitenant_utils.go:205,multitenant_upgrade.go:134,multitenant_upgrade.go:35,test_runner.go:896: cluster.RunE: context canceled
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1948
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1929
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:144
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:838
		  | main.(*testImpl).Fatal
		  | 	main/pkg/cmd/roachtest/test_impl.go:254
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).start.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:210
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError.func1
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:69
		  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
		  | 	github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:207
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithin
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:57
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:40
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).start
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:205
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:134
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:35
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:896
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (2) cluster.RunE
		Wraps: (3) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-08-18T12:34:34Z

roachtest.multitenant-upgrade failed with artifacts on master @ 5c2c62ecf1bea60c807edc6b4da22d900ad4ae03:

		  | SQLSTATE: 42704
		  | Failed running "mt start-sql"
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 2. Command with error:
		  | ``````
		  | v22.1.5/cockroach mt start-sql --certs-dir certs --tenant-id=12 --http-addr 0.0.0.0:8012 --kv-addrs 10.142.0.63:26257 --sql-addr 10.142.0.116:20012 --log="file-defaults: {dir: 'logs/mt-12', exit-on-error: false}" --store=cockroach-data-mt-12
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

	cluster.go:1931,multitenant_utils.go:144,panic.go:838,test_impl.go:254,multitenant_utils.go:210,soon.go:69,retry.go:207,soon.go:75,soon.go:57,soon.go:40,multitenant_utils.go:205,multitenant_upgrade.go:134,multitenant_upgrade.go:35,test_runner.go:896: cluster.RunE: context canceled
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1948
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1929
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:144
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:838
		  | main.(*testImpl).Fatal
		  | 	main/pkg/cmd/roachtest/test_impl.go:254
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).start.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:210
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError.func1
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:69
		  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
		  | 	github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:207
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithin
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:57
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:40
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).start
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:205
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:134
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:35
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:896
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (2) cluster.RunE
		Wraps: (3) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-08-19T13:14:34Z

roachtest.multitenant-upgrade failed with artifacts on master @ a0d8839aa6164af81a9ebb140147d3baf5321287:

		  | SQLSTATE: 42704
		  | Failed running "mt start-sql"
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 2. Command with error:
		  | ``````
		  | v22.1.5/cockroach mt start-sql --certs-dir certs --tenant-id=12 --http-addr 0.0.0.0:8012 --kv-addrs 10.142.0.48:26257 --sql-addr 10.142.0.17:20012 --log="file-defaults: {dir: 'logs/mt-12', exit-on-error: false}" --store=cockroach-data-mt-12
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

	cluster.go:1931,multitenant_utils.go:144,panic.go:838,test_impl.go:254,multitenant_utils.go:210,soon.go:69,retry.go:207,soon.go:75,soon.go:57,soon.go:40,multitenant_utils.go:205,multitenant_upgrade.go:134,multitenant_upgrade.go:35,test_runner.go:896: cluster.RunE: context canceled
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1948
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1929
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:144
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:838
		  | main.(*testImpl).Fatal
		  | 	main/pkg/cmd/roachtest/test_impl.go:254
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).start.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:210
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError.func1
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:69
		  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
		  | 	github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:207
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithin
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:57
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:40
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).start
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:205
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:134
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:35
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:896
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (2) cluster.RunE
		Wraps: (3) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-08-20T12:51:07Z

roachtest.multitenant-upgrade failed with artifacts on master @ cb55144cdec54d2a70f074ad64b4eca5e6c6891a:

		  | SQLSTATE: 42704
		  | Failed running "mt start-sql"
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 2. Command with error:
		  | ``````
		  | v22.1.5/cockroach mt start-sql --certs-dir certs --tenant-id=12 --http-addr 0.0.0.0:8012 --kv-addrs 10.142.0.26:26257 --sql-addr 10.142.0.153:20012 --log="file-defaults: {dir: 'logs/mt-12', exit-on-error: false}" --store=cockroach-data-mt-12
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

	cluster.go:1931,multitenant_utils.go:144,panic.go:838,test_impl.go:254,multitenant_utils.go:210,soon.go:69,retry.go:207,soon.go:75,soon.go:57,soon.go:40,multitenant_utils.go:205,multitenant_upgrade.go:134,multitenant_upgrade.go:35,test_runner.go:896: cluster.RunE: context canceled
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1948
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1929
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:144
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:838
		  | main.(*testImpl).Fatal
		  | 	main/pkg/cmd/roachtest/test_impl.go:254
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).start.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:210
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError.func1
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:69
		  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
		  | 	github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:207
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithin
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:57
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:40
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).start
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:205
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:134
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:35
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:896
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (2) cluster.RunE
		Wraps: (3) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-08-21T13:38:42Z

roachtest.multitenant-upgrade failed with artifacts on master @ e6a7dc2f8ee39549e186bd05626c4c375b76fd04:

		  | SQLSTATE: 42704
		  | Failed running "mt start-sql"
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 2. Command with error:
		  | ``````
		  | v22.1.5/cockroach mt start-sql --certs-dir certs --tenant-id=12 --http-addr 0.0.0.0:8012 --kv-addrs 10.142.0.115:26257 --sql-addr 10.142.0.104:20012 --log="file-defaults: {dir: 'logs/mt-12', exit-on-error: false}" --store=cockroach-data-mt-12
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

	cluster.go:1931,multitenant_utils.go:144,panic.go:838,test_impl.go:254,multitenant_utils.go:210,soon.go:69,retry.go:207,soon.go:75,soon.go:57,soon.go:40,multitenant_utils.go:205,multitenant_upgrade.go:134,multitenant_upgrade.go:35,test_runner.go:896: cluster.RunE: context canceled
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1948
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1929
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:144
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:838
		  | main.(*testImpl).Fatal
		  | 	main/pkg/cmd/roachtest/test_impl.go:254
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).start.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:210
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError.func1
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:69
		  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
		  | 	github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:207
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithin
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:57
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:40
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).start
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:205
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:134
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:35
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:896
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (2) cluster.RunE
		Wraps: (3) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-08-22T13:02:12Z

roachtest.multitenant-upgrade failed with artifacts on master @ aaf50e920ceff3c2863ab96b9e3614b8434b70a8:

		  | SQLSTATE: 42704
		  | Failed running "mt start-sql"
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 2. Command with error:
		  | ``````
		  | v22.1.5/cockroach mt start-sql --certs-dir certs --tenant-id=12 --http-addr 0.0.0.0:8012 --kv-addrs 10.142.0.49:26257 --sql-addr 10.142.0.67:20012 --log="file-defaults: {dir: 'logs/mt-12', exit-on-error: false}" --store=cockroach-data-mt-12
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

	cluster.go:1931,multitenant_utils.go:144,panic.go:838,test_impl.go:254,multitenant_utils.go:210,soon.go:69,retry.go:207,soon.go:75,soon.go:57,soon.go:40,multitenant_utils.go:205,multitenant_upgrade.go:134,multitenant_upgrade.go:35,test_runner.go:896: cluster.RunE: context canceled
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1948
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1929
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:144
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:838
		  | main.(*testImpl).Fatal
		  | 	main/pkg/cmd/roachtest/test_impl.go:254
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).start.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:210
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError.func1
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:69
		  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
		  | 	github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:207
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithin
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:57
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:40
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).start
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:205
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:134
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:35
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:896
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (2) cluster.RunE
		Wraps: (3) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-08-23T13:22:44Z

roachtest.multitenant-upgrade failed with artifacts on master @ 80c274877a917580af62be6eb0cd48c8c7ae9c08:

		  | SQLSTATE: 42704
		  | Failed running "mt start-sql"
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 2. Command with error:
		  | ``````
		  | v22.1.5/cockroach mt start-sql --certs-dir certs --tenant-id=12 --http-addr 0.0.0.0:8012 --kv-addrs 10.142.0.254:26257 --sql-addr 10.142.0.21:20012 --log="file-defaults: {dir: 'logs/mt-12', exit-on-error: false}" --store=cockroach-data-mt-12
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

	cluster.go:1931,multitenant_utils.go:144,panic.go:838,test_impl.go:254,multitenant_utils.go:210,soon.go:69,retry.go:207,soon.go:75,soon.go:57,soon.go:40,multitenant_utils.go:205,multitenant_upgrade.go:134,multitenant_upgrade.go:35,test_runner.go:896: cluster.RunE: context canceled
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1948
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1929
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:144
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:838
		  | main.(*testImpl).Fatal
		  | 	main/pkg/cmd/roachtest/test_impl.go:254
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).start.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:210
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError.func1
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:69
		  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
		  | 	github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:207
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithin
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:57
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:40
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).start
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:205
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:134
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:35
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:896
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (2) cluster.RunE
		Wraps: (3) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-08-24T13:35:50Z

roachtest.multitenant-upgrade failed with artifacts on master @ 003c0360de8b64319b5f0f127b99be91dbdca8a3:

		  | SQLSTATE: 42704
		  | Failed running "mt start-sql"
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 2. Command with error:
		  | ``````
		  | v22.1.5/cockroach mt start-sql --certs-dir certs --tenant-id=12 --http-addr 0.0.0.0:8012 --kv-addrs 10.142.0.113:26257 --sql-addr 10.142.0.118:20012 --log="file-defaults: {dir: 'logs/mt-12', exit-on-error: false}" --store=cockroach-data-mt-12
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

	cluster.go:1931,multitenant_utils.go:144,panic.go:838,test_impl.go:254,multitenant_utils.go:210,soon.go:69,retry.go:207,soon.go:75,soon.go:57,soon.go:40,multitenant_utils.go:205,multitenant_upgrade.go:134,multitenant_upgrade.go:35,test_runner.go:896: cluster.RunE: context canceled
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1948
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1929
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:144
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:838
		  | main.(*testImpl).Fatal
		  | 	main/pkg/cmd/roachtest/test_impl.go:254
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).start.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:210
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError.func1
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:69
		  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
		  | 	github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:207
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithin
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:57
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:40
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).start
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:205
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:134
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:35
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:896
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (2) cluster.RunE
		Wraps: (3) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-08-25T13:20:42Z

roachtest.multitenant-upgrade failed with artifacts on master @ 524fd14da3fefcd849f44a835cc5f88f5dbdadcc:

		  | SQLSTATE: 42704
		  | Failed running "mt start-sql"
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 2. Command with error:
		  | ``````
		  | v22.1.6/cockroach mt start-sql --certs-dir certs --tenant-id=12 --http-addr 0.0.0.0:8012 --kv-addrs 10.142.0.211:26257 --sql-addr 10.142.0.171:20012 --log="file-defaults: {dir: 'logs/mt-12', exit-on-error: false}" --store=cockroach-data-mt-12
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

	cluster.go:1930,multitenant_utils.go:144,panic.go:838,test_impl.go:254,multitenant_utils.go:210,soon.go:69,retry.go:207,soon.go:75,soon.go:57,soon.go:40,multitenant_utils.go:205,multitenant_upgrade.go:134,multitenant_upgrade.go:35,test_runner.go:896: cluster.RunE: context canceled
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1947
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1928
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:144
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:838
		  | main.(*testImpl).Fatal
		  | 	main/pkg/cmd/roachtest/test_impl.go:254
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).start.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:210
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError.func1
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:69
		  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
		  | 	github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:207
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithin
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:57
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:40
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).start
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:205
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:134
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:35
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:896
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (2) cluster.RunE
		Wraps: (3) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-08-27T13:42:52Z

roachtest.multitenant-upgrade failed with artifacts on master @ 770ff3c545a51752490403da64d56fb397f49c5e:

		  | SQLSTATE: 42704
		  | Failed running "mt start-sql"
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 2. Command with error:
		  | ``````
		  | v22.1.6/cockroach mt start-sql --certs-dir certs --tenant-id=12 --http-addr 0.0.0.0:8012 --kv-addrs 10.142.0.14:26257 --sql-addr 10.142.0.12:20012 --log="file-defaults: {dir: 'logs/mt-12', exit-on-error: false}" --store=cockroach-data-mt-12
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

	cluster.go:1940,multitenant_utils.go:144,panic.go:838,test_impl.go:259,multitenant_utils.go:210,soon.go:69,retry.go:207,soon.go:75,soon.go:57,soon.go:40,multitenant_utils.go:205,multitenant_upgrade.go:134,multitenant_upgrade.go:35,test_runner.go:897: cluster.RunE: context canceled
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1957
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1938
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:144
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:838
		  | main.(*testImpl).Fatal
		  | 	main/pkg/cmd/roachtest/test_impl.go:259
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).start.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:210
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError.func1
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:69
		  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
		  | 	github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:207
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithin
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:57
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:40
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).start
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:205
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:134
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:35
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:897
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (2) cluster.RunE
		Wraps: (3) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-08-28T13:54:31Z

roachtest.multitenant-upgrade failed with artifacts on master @ f59620ec646d1181d358d0dc41ab60815ecf59c9:

		  | SQLSTATE: 42704
		  | Failed running "mt start-sql"
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 2. Command with error:
		  | ``````
		  | v22.1.6/cockroach mt start-sql --certs-dir certs --tenant-id=12 --http-addr 0.0.0.0:8012 --kv-addrs 10.142.0.3:26257 --sql-addr 10.142.0.45:20012 --log="file-defaults: {dir: 'logs/mt-12', exit-on-error: false}" --store=cockroach-data-mt-12
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

	cluster.go:1940,multitenant_utils.go:144,panic.go:838,test_impl.go:259,multitenant_utils.go:210,soon.go:69,retry.go:207,soon.go:75,soon.go:57,soon.go:40,multitenant_utils.go:205,multitenant_upgrade.go:134,multitenant_upgrade.go:35,test_runner.go:897: cluster.RunE: context canceled
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1957
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1938
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:144
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:838
		  | main.(*testImpl).Fatal
		  | 	main/pkg/cmd/roachtest/test_impl.go:259
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).start.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:210
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError.func1
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:69
		  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
		  | 	github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:207
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithin
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:57
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:40
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).start
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:205
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:134
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:35
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:897
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (2) cluster.RunE
		Wraps: (3) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-08-29T12:54:20Z

roachtest.multitenant-upgrade failed with artifacts on master @ 3b16435371a43d603d193a1e2b480a23fba3f07a:

		  | SQLSTATE: 42704
		  | Failed running "mt start-sql"
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 2. Command with error:
		  | ``````
		  | v22.1.6/cockroach mt start-sql --certs-dir certs --tenant-id=12 --http-addr 0.0.0.0:8012 --kv-addrs 10.142.0.246:26257 --sql-addr 10.142.0.245:20012 --log="file-defaults: {dir: 'logs/mt-12', exit-on-error: false}" --store=cockroach-data-mt-12
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

	cluster.go:1940,multitenant_utils.go:144,panic.go:838,test_impl.go:259,multitenant_utils.go:210,soon.go:69,retry.go:207,soon.go:75,soon.go:57,soon.go:40,multitenant_utils.go:205,multitenant_upgrade.go:134,multitenant_upgrade.go:35,test_runner.go:897: cluster.RunE: context canceled
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1957
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1938
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:144
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:838
		  | main.(*testImpl).Fatal
		  | 	main/pkg/cmd/roachtest/test_impl.go:259
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).start.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:210
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError.func1
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:69
		  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
		  | 	github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:207
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithin
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:57
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:40
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).start
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:205
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:134
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:35
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:897
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (2) cluster.RunE
		Wraps: (3) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

rafiss · 2022-08-29T19:38:05Z

Currently this is failing because it is running on v22.1.6, which includes this change #85719. I think the test needs to be updated to account for that. I'll work on that now.

rafiss · 2022-08-29T19:45:23Z

Oh, I misread. the error is:

		Wraps: (3) v22.1.6/cockroach mt start-sql --certs-dir certs --tenant-id=12 --http-addr 0.0.0.0:8012 --kv-addrs 10.142.0.246:26257 --sql-addr 10.142.0.245:20012 --log="file-defaults: {dir: 'logs/mt-12', exit-on-error: false}" --store=cockroach-data-mt-12 returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | igrations/migrations.go:404  [nsql1] 8 +	: addRootUser: null value in column "user_id" violates not-null constraint
		  | W220829 12:54:04.959454 1 startupmigrations/migrations.go:404  [nsql1] 9  failed to run 
		  | W220829 12:54:04.959454 1 startupmigrations/migrations.go:404  [nsql1] 9 +          UPSERT INTO system.users (username, "hashedPassword", "isRole") VALUES ($1, '', true): addAdminRole: null value in column "user_id" violates not-null constraint
		  | W220829 12:54:05.012130 1 startupmigrations/migrations.go:404  [nsql1] 10  failed to run 
		  | W220829 12:54:05.012130 1 startupmigrations/migrations.go:404  [nsql1] 10 +          UPSERT INTO system.users (username, "hashedPassword", "isRole") VALUES ($1, '', true): addAdminRole: null value in column "user_id" violates not-null constraint
		  | W220829 12:54:05.129365 1 startupmigrations/migrations.go:404  [nsql1] 11  failed to run 
		  | W220829 12:54:05.129365 1 startupmigrations/migrations.go:404  [nsql1] 11 +          UPSERT INTO system.users (username, "hashedPassword", "isRole") VALUES ($1, '', true): addAdminRole: null value in column "user_id" violates not-null constraint
		  | W220829 12:54:05.320657 1 startupmigrations/migrations.go:404  [nsql1] 12  failed to run 
		  | W220829 12:54:05.320657 1 startupmigrations/migrations.go:404  [nsql1] 12 +          UPSERT INTO system.users (username, "hashedPassword", "isRole") VALUES ($1, '', true): addAdminRole: null value in column "user_id" violates not-null constraint
		  | W220829 12:54:05.713119 1 startupmigrations/migrations.go:404  [nsql1] 13  failed to run 
		  | W220829 12:54:05.713119 1 startupmigrations/migrations.go:404  [nsql1] 13 +          UPSERT INTO system.users (username, "hashedPassword", "isRole") VALUES ($1, '', true): addAdminRole: null value in column "user_id" violates not-null constraint
		  | W220829 12:54:06.573744 1 startupmigrations/migrations.go:404  [nsql1] 14  failed to run 
		  | W220829 12:54:06.573744 1 startupmigrations/migrations.go:404  [nsql1] 14 +          UPSERT INTO system.users (username, "hashedPassword", "isRole") VALUES ($1, '', true): addAdminRole: null value in column "user_id" violates not-null constraint
		  | W220829 12:54:06.606926 1 startupmigrations/migrations.go:856  [nsql1] 15  failed attempt to add database "defaultdb": create-default-DB: role/user "root" does not exist
		  | W220829 12:54:06.663248 1 startupmigrations/migrations.go:856  [nsql1] 16  failed attempt to add database "defaultdb": create-default-DB: role/user "root" does not exist
		  | W220829 12:54:06.767505 1 startupmigrations/migrations.go:856  [nsql1] 17  failed attempt to add database "defaultdb": create-default-DB: role/user "root" does not exist
		  | W220829 12:54:06.964707 1 startupmigrations/migrations.go:856  [nsql1] 18  failed attempt to add database "defaultdb": create-default-DB: role/user "root" does not exist
		  | W220829 12:54:07.334571 1 startupmigrations/migrations.go:856  [nsql1] 19  failed attempt to add database "defaultdb": create-default-DB: role/user "root" does not exist
		  | W220829 12:54:08.245512 1 startupmigrations/migrations.go:856  [nsql1] 20  failed attempt to add database "defaultdb": create-default-DB: role/user "root" does not exist
		  | W220829 12:54:08.247083 163 jobs/registry.go:811  [nsql1] 21  canceling all adopted jobs due to stopper quiescing
		  | W220829 12:54:08.247066 135 ccl/kvccl/kvtenantccl/connector.go:231  [nsql1,tenant-connector] 22  error consuming GossipSubscription RPC: rpc error: code = Canceled desc = context canceled
		  | W220829 12:54:08.247223 136 ccl/kvccl/kvtenantccl/setting_overrides.go:46  [nsql1,tenant-connector] 23  error consuming TenantSettings RPC: rpc error: code = Canceled desc = context canceled
		  | W220829 12:54:08.247261 61 sql/sqlliveness/slinstance/slinstance.go:235  [nsql1] 24  exiting heartbeat loop
		  | W220829 12:54:08.247830 215 kv/txn.go:728  [nsql1] 25  failure aborting transaction: node unavailable; try another peer; abort caused by: log-job: context canceled
		  | ERROR: ensuring SQL migrations: failed to run migration "create default databases": create-default-DB: role/user "root" does not exist
		  | SQLSTATE: 42704
		  | Failed running "mt start-sql"

cc @RichardJCai that seems related to #81457. could there be a problem with that in mixed-version multitenant clusters?

cockroach-teamcity · 2022-08-30T13:07:06Z

roachtest.multitenant-upgrade failed with artifacts on master @ d35c174b71065264b8a3910df3f488d10741f788:

		  | SQLSTATE: 42704
		  | Failed running "mt start-sql"
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 2. Command with error:
		  | ``````
		  | v22.1.6/cockroach mt start-sql --certs-dir certs --tenant-id=12 --http-addr 0.0.0.0:8012 --kv-addrs 10.142.0.31:26257 --sql-addr 10.142.0.51:20012 --log="file-defaults: {dir: 'logs/mt-12', exit-on-error: false}" --store=cockroach-data-mt-12
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

	cluster.go:1940,multitenant_utils.go:144,panic.go:838,test_impl.go:259,multitenant_utils.go:210,soon.go:69,retry.go:207,soon.go:75,soon.go:57,soon.go:40,multitenant_utils.go:205,multitenant_upgrade.go:134,multitenant_upgrade.go:35,test_runner.go:897: cluster.RunE: context canceled
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1957
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1938
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:144
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:838
		  | main.(*testImpl).Fatal
		  | 	main/pkg/cmd/roachtest/test_impl.go:259
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).start.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:210
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError.func1
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:69
		  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
		  | 	github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:207
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithin
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:57
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:40
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).start
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:205
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:134
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:35
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:897
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (2) cluster.RunE
		Wraps: (3) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

rafiss · 2022-08-30T21:32:06Z

Bisecting shows that the test started failing with #80353

The error is

error executing 'SELECT * FROM foo LIMIT 1': pq: failed to connect to n1 at 10.142.0.67:20011: initial connection heartbeat failed: rpc error: code = Unknown desc = client requested node ID 1 doesn't match server node ID 2

It's failing at step 11 of the below sequence.

1. start system tenant on v22.1 binary
2. run crdb_internal.create_tenant(11) and crdb_internal.create_tenant(12)
3. start tenant 11 on v22.1 binary
4. set preserve_downgrade_option on system tenant
5. restart system tenant on v22.2 binary
6. verify tenant 11 still works
7. start tenant 12 on v22.1 binary
8. run crdb_internal.create_tenant(13)
9. start tenant 13 on v22.2 binary
10. stop tenant 11 and start again using v22.2 binary (without upgrading tenant 11 cluster version)
11. run `SELECT * FROM foo LIMIT 1` on tenant 11

I also noticed that the test needs to be updated so that it does the version upgrades in the correct order. I'll make those changes, but we still have the problem described above.

yuzefovich · 2022-08-31T00:00:28Z

I looked over the code in #80353, and I don't think that PR itself introduced a bug - rather a bug was already present but hidden, and #80353 exposed it.

cockroach-teamcity · 2022-08-31T13:08:23Z

roachtest.multitenant-upgrade failed with artifacts on master @ 448352ce4ed71a58e7701e82c786dd5152498310:

		  | SQLSTATE: 42704
		  | Failed running "mt start-sql"
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 2. Command with error:
		  | ``````
		  | v22.1.6/cockroach mt start-sql --certs-dir certs --tenant-id=12 --http-addr 0.0.0.0:8012 --kv-addrs 10.142.0.109:26257 --sql-addr 10.142.0.121:20012 --log="file-defaults: {dir: 'logs/mt-12', exit-on-error: false}" --store=cockroach-data-mt-12
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

	cluster.go:1940,multitenant_utils.go:144,panic.go:838,test_impl.go:259,multitenant_utils.go:210,soon.go:69,retry.go:207,soon.go:75,soon.go:57,soon.go:40,multitenant_utils.go:205,multitenant_upgrade.go:134,multitenant_upgrade.go:35,test_runner.go:897: cluster.RunE: context canceled
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1957
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1938
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:144
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:838
		  | main.(*testImpl).Fatal
		  | 	main/pkg/cmd/roachtest/test_impl.go:259
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).start.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:210
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError.func1
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:69
		  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
		  | 	github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:207
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithin
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:57
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:40
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).start
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:205
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:134
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:35
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:897
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (2) cluster.RunE
		Wraps: (3) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

rafiss · 2022-08-31T16:11:50Z

I added some additional logs - it looks like the request is from tenant 11 heartbeating to itself, using a stale sqlInstanceID. so maybe that means the old sqlInstanceID was not properly cleaned up when tenant 11 was restarted:

client requested node ID 1 doesn't match server node ID 2

PingRequest{
  origin_addr:"10.142.0.16:20011" 
  target_node_id:1
  origin_node_id:2
}

So I'll look into the tenant 11 shutdown next. However, I think even in the case of unclean shutdown of a tenant pod, we wouldn't want this kind of failure mode.

rafiss · 2022-09-01T04:19:15Z

I found the problem; the sqlinstance cache was not being updated with the correct timestamps. The fix is in #87111

cockroach-teamcity · 2022-09-01T13:44:10Z

roachtest.multitenant-upgrade failed with artifacts on master @ b316a5ed5fe7253d113174d9d95ddebf1143b4e4:

		  | SQLSTATE: 42704
		  | Failed running "mt start-sql"
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 2. Command with error:
		  | ``````
		  | v22.1.6/cockroach mt start-sql --certs-dir certs --tenant-id=12 --http-addr 0.0.0.0:8012 --kv-addrs 10.142.0.133:26257 --sql-addr 10.142.0.33:20012 --log="file-defaults: {dir: 'logs/mt-12', exit-on-error: false}" --store=cockroach-data-mt-12
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

	cluster.go:1940,multitenant_utils.go:144,panic.go:838,test_impl.go:259,multitenant_utils.go:210,soon.go:69,retry.go:207,soon.go:75,soon.go:57,soon.go:40,multitenant_utils.go:205,multitenant_upgrade.go:134,multitenant_upgrade.go:35,test_runner.go:897: cluster.RunE: context canceled
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1957
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1938
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).stop
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:144
		  | runtime.gopanic
		  | 	GOROOT/src/runtime/panic.go:838
		  | main.(*testImpl).Fatal
		  | 	main/pkg/cmd/roachtest/test_impl.go:259
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).start.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:210
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError.func1
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:69
		  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
		  | 	github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:207
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithin
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:57
		  | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon
		  | 	github.com/cockroachdb/cockroach/pkg/testutils/soon.go:40
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).start
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:205
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:134
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:35
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:897
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (2) cluster.RunE
		Wraps: (3) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

blathers-crl bot added the T-kv KV Team label May 19, 2022

jlinder added sync-me and removed sync-me labels May 20, 2022

lidorcarmel added the S-1 High impact: many users impacted, serious risk of high unavailability or data loss label May 24, 2022

cockroach-teamcity mentioned this issue May 31, 2022

roachtest: multitenant-upgrade failed #82122

Closed

ajstorm assigned rafiss Aug 24, 2022

rafiss mentioned this issue Aug 30, 2022

roachtest,sqlinstance: fix multitenant_upgrade test #87111

Merged

craig bot closed this as completed in d18ecbb Sep 1, 2022

roachtest: multitenant-upgrade failed #81517

roachtest: multitenant-upgrade failed #81517

Comments

cockroach-teamcity commented May 19, 2022 • edited by cockroach-jira-scripts Loading

cockroach-teamcity commented May 21, 2022

lidorcarmel commented May 24, 2022

tbg commented May 24, 2022

cockroach-teamcity commented May 24, 2022

cockroach-teamcity commented May 25, 2022

cockroach-teamcity commented May 26, 2022

cockroach-teamcity commented May 27, 2022

cockroach-teamcity commented May 28, 2022

cockroach-teamcity commented May 29, 2022

cockroach-teamcity commented May 30, 2022

cockroach-teamcity commented May 31, 2022

cockroach-teamcity commented Jun 1, 2022

tbg commented Jun 2, 2022

cockroach-teamcity commented Jun 2, 2022

cockroach-teamcity commented Jun 3, 2022

cockroach-teamcity commented Jun 4, 2022

cockroach-teamcity commented Jun 5, 2022

cockroach-teamcity commented Jun 6, 2022

cockroach-teamcity commented Aug 13, 2022

cockroach-teamcity commented Aug 14, 2022

cockroach-teamcity commented Aug 15, 2022

cockroach-teamcity commented Aug 16, 2022

cockroach-teamcity commented Aug 17, 2022

cockroach-teamcity commented Aug 18, 2022

cockroach-teamcity commented Aug 19, 2022

cockroach-teamcity commented Aug 20, 2022

cockroach-teamcity commented Aug 21, 2022

cockroach-teamcity commented Aug 22, 2022

cockroach-teamcity commented Aug 23, 2022

cockroach-teamcity commented Aug 24, 2022

cockroach-teamcity commented Aug 25, 2022

cockroach-teamcity commented Aug 27, 2022

cockroach-teamcity commented Aug 28, 2022

cockroach-teamcity commented Aug 29, 2022

rafiss commented Aug 29, 2022

rafiss commented Aug 29, 2022

cockroach-teamcity commented Aug 30, 2022

rafiss commented Aug 30, 2022

yuzefovich commented Aug 31, 2022

cockroach-teamcity commented Aug 31, 2022

rafiss commented Aug 31, 2022

rafiss commented Sep 1, 2022

cockroach-teamcity commented Sep 1, 2022

cockroach-teamcity commented May 19, 2022 •

edited by cockroach-jira-scripts

Loading