roachtest: multitenant-upgrade failed #85211

cockroach-teamcity · 2022-07-28T11:10:19Z

roachtest.multitenant-upgrade failed with artifacts on release-22.1 @ 7d74b25b21bdabc3b0cc87ced447acc9967eb4d5:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/multitenant-upgrade/run_1
	cluster.go:1934,multitenant_utils.go:133,multitenant_utils.go:85,multitenant_upgrade.go:89,multitenant_upgrade.go:35,test_runner.go:883: output in run_111004.267631447_n2_cockroach_cert: ./cockroach cert create-client root --certs-dir=certs --ca-key=certs/ca.key --tenant-scope 1,11,11,12,13,14 --overwrite returned: COMMAND_PROBLEM: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1968
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1932
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).recreateClientCertsWithTenantScope
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:133
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.createTenantNode
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:85
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:89
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:35
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:883
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_111004.267631447_n2_cockroach_cert
		Wraps: (3) ./cockroach cert create-client root --certs-dir=certs --ca-key=certs/ca.key --tenant-scope 1,11,11,12,13,14 --overwrite returned
		  | stderr:
		  | ERROR: connection lost.
		  |
		  | failed to generate client certificate and key: error loading CA certificate certs/ca.crt and key certs/ca.key: open certs/ca.crt: no such file or directory
		  | Failed running "cert create-client"
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 2. Command with error:
		  | ``````
		  | ./cockroach cert create-client root --certs-dir=certs --ca-key=certs/ca.key --tenant-scope 1,11,11,12,13,14 --overwrite
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: multitenant-upgrade failed #81517 roachtest: multitenant-upgrade failed [A-multitenancy C-test-failure O-roachtest O-robot S-1 T-kv branch-master release-blocker sync-me-8]

/cc @cockroachdb/multi-tenant _{This test on roachdash | Improve this report!

Jira issue: CRDB-18105}

The text was updated successfully, but these errors were encountered:

cockroach-teamcity · 2022-07-29T11:23:58Z

roachtest.multitenant-upgrade failed with artifacts on release-22.1 @ 6f5197c818619f616a5c8e3ddfedc923f1f6bd26:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/multitenant-upgrade/run_1
	cluster.go:1934,multitenant_utils.go:133,multitenant_utils.go:85,multitenant_upgrade.go:89,multitenant_upgrade.go:35,test_runner.go:883: output in run_112342.835772223_n2_cockroach_cert: ./cockroach cert create-client root --certs-dir=certs --ca-key=certs/ca.key --tenant-scope 1,11,11,12,13,14 --overwrite returned: COMMAND_PROBLEM: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1968
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1932
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).recreateClientCertsWithTenantScope
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:133
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.createTenantNode
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:85
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:89
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:35
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:883
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_112342.835772223_n2_cockroach_cert
		Wraps: (3) ./cockroach cert create-client root --certs-dir=certs --ca-key=certs/ca.key --tenant-scope 1,11,11,12,13,14 --overwrite returned
		  | stderr:
		  | ERROR: connection lost.
		  |
		  | failed to generate client certificate and key: error loading CA certificate certs/ca.crt and key certs/ca.key: open certs/ca.crt: no such file or directory
		  | Failed running "cert create-client"
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 2. Command with error:
		  | ``````
		  | ./cockroach cert create-client root --certs-dir=certs --ca-key=certs/ca.key --tenant-scope 1,11,11,12,13,14 --overwrite
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: multitenant-upgrade failed #81517 roachtest: multitenant-upgrade failed [A-multitenancy C-test-failure O-roachtest O-robot S-1 T-kv branch-master release-blocker sync-me-8]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-07-30T11:55:42Z

roachtest.multitenant-upgrade failed with artifacts on release-22.1 @ 60293f9fdb49de3c0e0a7ad4bf49f92cec05b1a5:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/multitenant-upgrade/run_1
	cluster.go:1934,multitenant_utils.go:133,multitenant_utils.go:85,multitenant_upgrade.go:89,multitenant_upgrade.go:35,test_runner.go:883: output in run_115526.687188179_n2_cockroach_cert: ./cockroach cert create-client root --certs-dir=certs --ca-key=certs/ca.key --tenant-scope 1,11,11,12,13,14 --overwrite returned: COMMAND_PROBLEM: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1968
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1932
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).recreateClientCertsWithTenantScope
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:133
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.createTenantNode
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:85
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:89
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:35
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:883
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_115526.687188179_n2_cockroach_cert
		Wraps: (3) ./cockroach cert create-client root --certs-dir=certs --ca-key=certs/ca.key --tenant-scope 1,11,11,12,13,14 --overwrite returned
		  | stderr:
		  | ERROR: connection lost.
		  |
		  | failed to generate client certificate and key: error loading CA certificate certs/ca.crt and key certs/ca.key: open certs/ca.crt: no such file or directory
		  | Failed running "cert create-client"
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 2. Command with error:
		  | ``````
		  | ./cockroach cert create-client root --certs-dir=certs --ca-key=certs/ca.key --tenant-scope 1,11,11,12,13,14 --overwrite
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: multitenant-upgrade failed #81517 roachtest: multitenant-upgrade failed [A-multitenancy C-test-failure O-roachtest O-robot S-1 T-kv branch-master release-blocker sync-me-8]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-07-31T11:21:28Z

roachtest.multitenant-upgrade failed with artifacts on release-22.1 @ 60293f9fdb49de3c0e0a7ad4bf49f92cec05b1a5:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/multitenant-upgrade/run_1
	cluster.go:1934,multitenant_utils.go:133,multitenant_utils.go:85,multitenant_upgrade.go:89,multitenant_upgrade.go:35,test_runner.go:883: output in run_112113.903616171_n2_cockroach_cert: ./cockroach cert create-client root --certs-dir=certs --ca-key=certs/ca.key --tenant-scope 1,11,11,12,13,14 --overwrite returned: COMMAND_PROBLEM: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1968
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1932
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).recreateClientCertsWithTenantScope
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:133
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.createTenantNode
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:85
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:89
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:35
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:883
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_112113.903616171_n2_cockroach_cert
		Wraps: (3) ./cockroach cert create-client root --certs-dir=certs --ca-key=certs/ca.key --tenant-scope 1,11,11,12,13,14 --overwrite returned
		  | stderr:
		  | ERROR: connection lost.
		  |
		  | failed to generate client certificate and key: error loading CA certificate certs/ca.crt and key certs/ca.key: open certs/ca.crt: no such file or directory
		  | Failed running "cert create-client"
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 2. Command with error:
		  | ``````
		  | ./cockroach cert create-client root --certs-dir=certs --ca-key=certs/ca.key --tenant-scope 1,11,11,12,13,14 --overwrite
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: multitenant-upgrade failed #81517 roachtest: multitenant-upgrade failed [A-multitenancy C-test-failure O-roachtest O-robot S-1 T-kv branch-master release-blocker sync-me-8]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-07-31T18:24:03Z

roachtest.multitenant-upgrade failed with artifacts on release-22.1 @ 60293f9fdb49de3c0e0a7ad4bf49f92cec05b1a5:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/multitenant-upgrade/run_1
	cluster.go:1934,multitenant_utils.go:133,multitenant_utils.go:85,multitenant_upgrade.go:89,multitenant_upgrade.go:35,test_runner.go:883: output in run_182347.931548724_n2_cockroach_cert: ./cockroach cert create-client root --certs-dir=certs --ca-key=certs/ca.key --tenant-scope 1,11,11,12,13,14 --overwrite returned: COMMAND_PROBLEM: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1968
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1932
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).recreateClientCertsWithTenantScope
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:133
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.createTenantNode
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:85
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:89
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:35
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:883
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_182347.931548724_n2_cockroach_cert
		Wraps: (3) ./cockroach cert create-client root --certs-dir=certs --ca-key=certs/ca.key --tenant-scope 1,11,11,12,13,14 --overwrite returned
		  | stderr:
		  | ERROR: connection lost.
		  |
		  | failed to generate client certificate and key: error loading CA certificate certs/ca.crt and key certs/ca.key: open certs/ca.crt: no such file or directory
		  | Failed running "cert create-client"
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 2. Command with error:
		  | ``````
		  | ./cockroach cert create-client root --certs-dir=certs --ca-key=certs/ca.key --tenant-scope 1,11,11,12,13,14 --overwrite
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: multitenant-upgrade failed #81517 roachtest: multitenant-upgrade failed [A-multitenancy C-test-failure O-roachtest O-robot S-1 T-kv branch-master release-blocker sync-me-8]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-08-01T11:25:45Z

roachtest.multitenant-upgrade failed with artifacts on release-22.1 @ 60293f9fdb49de3c0e0a7ad4bf49f92cec05b1a5:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/multitenant-upgrade/run_1
	cluster.go:1934,multitenant_utils.go:133,multitenant_utils.go:85,multitenant_upgrade.go:89,multitenant_upgrade.go:35,test_runner.go:883: output in run_112531.254150666_n2_cockroach_cert: ./cockroach cert create-client root --certs-dir=certs --ca-key=certs/ca.key --tenant-scope 1,11,11,12,13,14 --overwrite returned: COMMAND_PROBLEM: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1968
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1932
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).recreateClientCertsWithTenantScope
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:133
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.createTenantNode
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:85
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:89
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:35
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:883
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_112531.254150666_n2_cockroach_cert
		Wraps: (3) ./cockroach cert create-client root --certs-dir=certs --ca-key=certs/ca.key --tenant-scope 1,11,11,12,13,14 --overwrite returned
		  | stderr:
		  | ERROR: connection lost.
		  |
		  | failed to generate client certificate and key: error loading CA certificate certs/ca.crt and key certs/ca.key: open certs/ca.crt: no such file or directory
		  | Failed running "cert create-client"
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 2. Command with error:
		  | ``````
		  | ./cockroach cert create-client root --certs-dir=certs --ca-key=certs/ca.key --tenant-scope 1,11,11,12,13,14 --overwrite
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: multitenant-upgrade failed #81517 roachtest: multitenant-upgrade failed [A-multitenancy C-test-failure O-roachtest O-robot S-1 T-kv branch-master release-blocker sync-me-8]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-08-02T11:13:05Z

roachtest.multitenant-upgrade failed with artifacts on release-22.1 @ c7b53317f766d7bbdc24ea78f73c49cc21dbfa63:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/multitenant-upgrade/run_1
	cluster.go:1934,multitenant_utils.go:133,multitenant_utils.go:85,multitenant_upgrade.go:89,multitenant_upgrade.go:35,test_runner.go:883: output in run_111249.735920006_n2_cockroach_cert: ./cockroach cert create-client root --certs-dir=certs --ca-key=certs/ca.key --tenant-scope 1,11,11,12,13,14 --overwrite returned: COMMAND_PROBLEM: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1968
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1932
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).recreateClientCertsWithTenantScope
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:133
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.createTenantNode
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:85
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:89
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:35
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:883
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_111249.735920006_n2_cockroach_cert
		Wraps: (3) ./cockroach cert create-client root --certs-dir=certs --ca-key=certs/ca.key --tenant-scope 1,11,11,12,13,14 --overwrite returned
		  | stderr:
		  | ERROR: connection lost.
		  |
		  | failed to generate client certificate and key: error loading CA certificate certs/ca.crt and key certs/ca.key: open certs/ca.crt: no such file or directory
		  | Failed running "cert create-client"
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 2. Command with error:
		  | ``````
		  | ./cockroach cert create-client root --certs-dir=certs --ca-key=certs/ca.key --tenant-scope 1,11,11,12,13,14 --overwrite
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: multitenant-upgrade failed #81517 roachtest: multitenant-upgrade failed [A-multitenancy C-test-failure O-roachtest O-robot S-1 T-kv branch-master release-blocker sync-me-8]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-08-03T11:34:00Z

roachtest.multitenant-upgrade failed with artifacts on release-22.1 @ eae66256fdc8c6afeb8146d6d4daca25d9ad4118:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/multitenant-upgrade/run_1
	cluster.go:1934,multitenant_utils.go:133,multitenant_utils.go:85,multitenant_upgrade.go:89,multitenant_upgrade.go:35,test_runner.go:883: output in run_113345.216860443_n2_cockroach_cert: ./cockroach cert create-client root --certs-dir=certs --ca-key=certs/ca.key --tenant-scope 1,11,11,12,13,14 --overwrite returned: COMMAND_PROBLEM: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1968
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1932
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).recreateClientCertsWithTenantScope
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:133
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.createTenantNode
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:85
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:89
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:35
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:883
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_113345.216860443_n2_cockroach_cert
		Wraps: (3) ./cockroach cert create-client root --certs-dir=certs --ca-key=certs/ca.key --tenant-scope 1,11,11,12,13,14 --overwrite returned
		  | stderr:
		  | ERROR: connection lost.
		  |
		  | failed to generate client certificate and key: error loading CA certificate certs/ca.crt and key certs/ca.key: open certs/ca.crt: no such file or directory
		  | Failed running "cert create-client"
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 2. Command with error:
		  | ``````
		  | ./cockroach cert create-client root --certs-dir=certs --ca-key=certs/ca.key --tenant-scope 1,11,11,12,13,14 --overwrite
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: multitenant-upgrade failed #81517 roachtest: multitenant-upgrade failed [A-multitenancy C-test-failure O-roachtest O-robot S-1 T-kv branch-master release-blocker sync-me-8]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-08-03T19:19:43Z

roachtest.multitenant-upgrade failed with artifacts on release-22.1 @ eae66256fdc8c6afeb8146d6d4daca25d9ad4118:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/multitenant-upgrade/run_1
	cluster.go:1934,multitenant_utils.go:133,multitenant_utils.go:85,multitenant_upgrade.go:89,multitenant_upgrade.go:35,test_runner.go:883: output in run_191927.744153686_n2_cockroach_cert: ./cockroach cert create-client root --certs-dir=certs --ca-key=certs/ca.key --tenant-scope 1,11,11,12,13,14 --overwrite returned: COMMAND_PROBLEM: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1968
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1932
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).recreateClientCertsWithTenantScope
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:133
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.createTenantNode
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:85
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:89
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:35
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:883
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_191927.744153686_n2_cockroach_cert
		Wraps: (3) ./cockroach cert create-client root --certs-dir=certs --ca-key=certs/ca.key --tenant-scope 1,11,11,12,13,14 --overwrite returned
		  | stderr:
		  | ERROR: connection lost.
		  |
		  | failed to generate client certificate and key: error loading CA certificate certs/ca.crt and key certs/ca.key: open certs/ca.crt: no such file or directory
		  | Failed running "cert create-client"
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 2. Command with error:
		  | ``````
		  | ./cockroach cert create-client root --certs-dir=certs --ca-key=certs/ca.key --tenant-scope 1,11,11,12,13,14 --overwrite
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: multitenant-upgrade failed #81517 roachtest: multitenant-upgrade failed [A-multitenancy C-test-failure O-roachtest O-robot S-1 T-kv branch-master release-blocker sync-me-8]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-08-03T22:18:32Z

roachtest.multitenant-upgrade failed with artifacts on release-22.1 @ da1c029daa85869e16b19a73c228546c98830379:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/multitenant-upgrade/run_1
	cluster.go:1934,multitenant_utils.go:133,multitenant_utils.go:85,multitenant_upgrade.go:89,multitenant_upgrade.go:35,test_runner.go:883: output in run_221816.374916438_n2_cockroach_cert: ./cockroach cert create-client root --certs-dir=certs --ca-key=certs/ca.key --tenant-scope 1,11,11,12,13,14 --overwrite returned: COMMAND_PROBLEM: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1968
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1932
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).recreateClientCertsWithTenantScope
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:133
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.createTenantNode
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:85
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:89
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:35
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:883
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_221816.374916438_n2_cockroach_cert
		Wraps: (3) ./cockroach cert create-client root --certs-dir=certs --ca-key=certs/ca.key --tenant-scope 1,11,11,12,13,14 --overwrite returned
		  | stderr:
		  | ERROR: connection lost.
		  |
		  | failed to generate client certificate and key: error loading CA certificate certs/ca.crt and key certs/ca.key: open certs/ca.crt: no such file or directory
		  | Failed running "cert create-client"
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 2. Command with error:
		  | ``````
		  | ./cockroach cert create-client root --certs-dir=certs --ca-key=certs/ca.key --tenant-scope 1,11,11,12,13,14 --overwrite
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: multitenant-upgrade failed #81517 roachtest: multitenant-upgrade failed [A-multitenancy C-test-failure O-roachtest O-robot S-1 T-kv branch-master release-blocker sync-me-8]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-08-04T11:06:09Z

roachtest.multitenant-upgrade failed with artifacts on release-22.1 @ b924e9dd007645c7f0a258dca7771739d97c2fe7:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/multitenant-upgrade/run_1
	cluster.go:1934,multitenant_utils.go:133,multitenant_utils.go:85,multitenant_upgrade.go:89,multitenant_upgrade.go:35,test_runner.go:883: output in run_110554.003032159_n2_cockroach_cert: ./cockroach cert create-client root --certs-dir=certs --ca-key=certs/ca.key --tenant-scope 1,11,11,12,13,14 --overwrite returned: COMMAND_PROBLEM: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1968
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1932
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).recreateClientCertsWithTenantScope
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:133
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.createTenantNode
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:85
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:89
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:35
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:883
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_110554.003032159_n2_cockroach_cert
		Wraps: (3) ./cockroach cert create-client root --certs-dir=certs --ca-key=certs/ca.key --tenant-scope 1,11,11,12,13,14 --overwrite returned
		  | stderr:
		  | ERROR: connection lost.
		  |
		  | failed to generate client certificate and key: error loading CA certificate certs/ca.crt and key certs/ca.key: open certs/ca.crt: no such file or directory
		  | Failed running "cert create-client"
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 2. Command with error:
		  | ``````
		  | ./cockroach cert create-client root --certs-dir=certs --ca-key=certs/ca.key --tenant-scope 1,11,11,12,13,14 --overwrite
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: multitenant-upgrade failed #81517 roachtest: multitenant-upgrade failed [A-multitenancy C-test-failure O-roachtest O-robot S-1 T-kv branch-master release-blocker sync-me-8]

_{This test on roachdash | Improve this report!}

rafiss · 2022-08-04T18:10:35Z

cc @abarganier - i think the backport from #84371 didn't fully work, since this test has been failing since the backport was merged.

cockroach-teamcity · 2022-08-05T11:45:20Z

roachtest.multitenant-upgrade failed with artifacts on release-22.1 @ 64466efaeeed24c8ef3ba4555cd1f88f18bfe2ba:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/multitenant-upgrade/run_1
	cluster.go:1934,multitenant_utils.go:133,multitenant_utils.go:85,multitenant_upgrade.go:89,multitenant_upgrade.go:35,test_runner.go:883: output in run_114505.778544588_n2_cockroach_cert: ./cockroach cert create-client root --certs-dir=certs --ca-key=certs/ca.key --tenant-scope 1,11,11,12,13,14 --overwrite returned: COMMAND_PROBLEM: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1968
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1932
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).recreateClientCertsWithTenantScope
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:133
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.createTenantNode
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:85
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:89
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:35
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:883
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_114505.778544588_n2_cockroach_cert
		Wraps: (3) ./cockroach cert create-client root --certs-dir=certs --ca-key=certs/ca.key --tenant-scope 1,11,11,12,13,14 --overwrite returned
		  | stderr:
		  | ERROR: connection lost.
		  |
		  | failed to generate client certificate and key: error loading CA certificate certs/ca.crt and key certs/ca.key: open certs/ca.crt: no such file or directory
		  | Failed running "cert create-client"
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 2. Command with error:
		  | ``````
		  | ./cockroach cert create-client root --certs-dir=certs --ca-key=certs/ca.key --tenant-scope 1,11,11,12,13,14 --overwrite
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: multitenant-upgrade failed #81517 roachtest: multitenant-upgrade failed [A-multitenancy C-test-failure O-roachtest O-robot S-1 T-kv branch-master release-blocker sync-me-8]

_{This test on roachdash | Improve this report!}

abarganier · 2022-08-05T14:30:40Z

@rafiss thanks for the CC - I'll take a look

cockroach-teamcity · 2022-08-06T11:09:08Z

roachtest.multitenant-upgrade failed with artifacts on release-22.1 @ 2d8002a0549c09beb566ae18533e164ed824f2ce:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/multitenant-upgrade/run_1
	cluster.go:1934,multitenant_utils.go:133,multitenant_utils.go:85,multitenant_upgrade.go:89,multitenant_upgrade.go:35,test_runner.go:883: output in run_110853.261781168_n2_cockroach_cert: ./cockroach cert create-client root --certs-dir=certs --ca-key=certs/ca.key --tenant-scope 1,11,11,12,13,14 --overwrite returned: COMMAND_PROBLEM: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1968
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1932
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).recreateClientCertsWithTenantScope
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:133
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.createTenantNode
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:85
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:89
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:35
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:883
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_110853.261781168_n2_cockroach_cert
		Wraps: (3) ./cockroach cert create-client root --certs-dir=certs --ca-key=certs/ca.key --tenant-scope 1,11,11,12,13,14 --overwrite returned
		  | stderr:
		  | ERROR: connection lost.
		  |
		  | failed to generate client certificate and key: error loading CA certificate certs/ca.crt and key certs/ca.key: open certs/ca.crt: no such file or directory
		  | Failed running "cert create-client"
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 2. Command with error:
		  | ``````
		  | ./cockroach cert create-client root --certs-dir=certs --ca-key=certs/ca.key --tenant-scope 1,11,11,12,13,14 --overwrite
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: multitenant-upgrade failed #81517 roachtest: multitenant-upgrade failed [A-multitenancy C-test-failure O-roachtest O-robot S-1 T-kv branch-master release-blocker sync-me-8]

_{This test on roachdash | Improve this report!}

abarganier · 2022-08-06T14:37:31Z

Okay, I found a solution. Thankfully, this is a roachprod issue, not an issue with the tenant certificates themselves.

When backporting tenant-scoped client certificates (#84371), we missed some necessary roachprod changes to setup tenant certificate bundles on roachprod nodes to support running tenants in secure mode.

The fix is to backport #82248 to support roachtests running tenants in secure mode. I've verified that pulling this PR into the release-22.1 branch allows multitenant-upgrade to pass, and as an added bonus, we no longer have to skip the acceptance/multitenant roachtest anymore on release-22.1.

I'll have a backport up & merged by EOD Monday. I am confident we can resolve this release blocker in time for the v22.1.6 release's branch cutoff (at the time of writing, August 10th).

cockroach-teamcity · 2022-08-07T11:29:33Z

roachtest.multitenant-upgrade failed with artifacts on release-22.1 @ 2d8002a0549c09beb566ae18533e164ed824f2ce:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/multitenant-upgrade/run_1
	cluster.go:1934,multitenant_utils.go:133,multitenant_utils.go:85,multitenant_upgrade.go:89,multitenant_upgrade.go:35,test_runner.go:883: output in run_112918.272372450_n2_cockroach_cert: ./cockroach cert create-client root --certs-dir=certs --ca-key=certs/ca.key --tenant-scope 1,11,11,12,13,14 --overwrite returned: COMMAND_PROBLEM: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1968
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1932
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).recreateClientCertsWithTenantScope
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:133
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.createTenantNode
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:85
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:89
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:35
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:883
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_112918.272372450_n2_cockroach_cert
		Wraps: (3) ./cockroach cert create-client root --certs-dir=certs --ca-key=certs/ca.key --tenant-scope 1,11,11,12,13,14 --overwrite returned
		  | stderr:
		  | ERROR: connection lost.
		  |
		  | failed to generate client certificate and key: error loading CA certificate certs/ca.crt and key certs/ca.key: open certs/ca.crt: no such file or directory
		  | Failed running "cert create-client"
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 2. Command with error:
		  | ``````
		  | ./cockroach cert create-client root --certs-dir=certs --ca-key=certs/ca.key --tenant-scope 1,11,11,12,13,14 --overwrite
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: multitenant-upgrade failed #81517 roachtest: multitenant-upgrade failed [A-multitenancy C-test-failure O-roachtest O-robot S-1 T-kv branch-master release-blocker sync-me-8]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-08-08T11:45:46Z

roachtest.multitenant-upgrade failed with artifacts on release-22.1 @ 2d8002a0549c09beb566ae18533e164ed824f2ce:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/multitenant-upgrade/run_1
	cluster.go:1934,multitenant_utils.go:133,multitenant_utils.go:85,multitenant_upgrade.go:89,multitenant_upgrade.go:35,test_runner.go:883: output in run_114531.116444741_n2_cockroach_cert: ./cockroach cert create-client root --certs-dir=certs --ca-key=certs/ca.key --tenant-scope 1,11,11,12,13,14 --overwrite returned: COMMAND_PROBLEM: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1968
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1932
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tenantNode).recreateClientCertsWithTenantScope
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:133
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.createTenantNode
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_utils.go:85
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runMultiTenantUpgrade
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:89
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerMultiTenantUpgrade.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/multitenant_upgrade.go:35
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:883
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_114531.116444741_n2_cockroach_cert
		Wraps: (3) ./cockroach cert create-client root --certs-dir=certs --ca-key=certs/ca.key --tenant-scope 1,11,11,12,13,14 --overwrite returned
		  | stderr:
		  | ERROR: connection lost.
		  |
		  | failed to generate client certificate and key: error loading CA certificate certs/ca.crt and key certs/ca.key: open certs/ca.crt: no such file or directory
		  | Failed running "cert create-client"
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 2. Command with error:
		  | ``````
		  | ./cockroach cert create-client root --certs-dir=certs --ca-key=certs/ca.key --tenant-scope 1,11,11,12,13,14 --overwrite
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: multitenant-upgrade failed #81517 roachtest: multitenant-upgrade failed [A-multitenancy C-test-failure O-roachtest O-robot S-1 T-kv branch-master release-blocker sync-me-8]

_{This test on roachdash | Improve this report!}

abarganier · 2022-08-08T16:26:32Z

Solved via #85719, marking as closed.

cockroach-teamcity added this to the 22.1 milestone Jul 28, 2022

cockroach-teamcity mentioned this issue Jul 28, 2022

roachtest: multitenant-upgrade failed #81517

Closed

rafiss assigned abarganier Aug 4, 2022

abarganier mentioned this issue Aug 7, 2022

release-22.1: roachprod: support --secure flag for start-tenant #85719

Merged

abarganier added the T-observability-inf label Aug 7, 2022

blathers-crl bot added the A-observability-inf label Aug 7, 2022

abarganier closed this as completed Aug 8, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

roachtest: multitenant-upgrade failed #85211

roachtest: multitenant-upgrade failed #85211

cockroach-teamcity commented Jul 28, 2022 •

edited by cockroach-jira-scripts

Loading

cockroach-teamcity commented Jul 29, 2022

cockroach-teamcity commented Jul 30, 2022

cockroach-teamcity commented Jul 31, 2022

cockroach-teamcity commented Jul 31, 2022

cockroach-teamcity commented Aug 1, 2022

cockroach-teamcity commented Aug 2, 2022

cockroach-teamcity commented Aug 3, 2022

cockroach-teamcity commented Aug 3, 2022

cockroach-teamcity commented Aug 3, 2022

cockroach-teamcity commented Aug 4, 2022

rafiss commented Aug 4, 2022

cockroach-teamcity commented Aug 5, 2022

abarganier commented Aug 5, 2022

cockroach-teamcity commented Aug 6, 2022

abarganier commented Aug 6, 2022 •

edited

Loading

cockroach-teamcity commented Aug 7, 2022

cockroach-teamcity commented Aug 8, 2022

abarganier commented Aug 8, 2022

roachtest: multitenant-upgrade failed #85211

roachtest: multitenant-upgrade failed #85211

Comments

cockroach-teamcity commented Jul 28, 2022 • edited by cockroach-jira-scripts Loading

cockroach-teamcity commented Jul 29, 2022

cockroach-teamcity commented Jul 30, 2022

cockroach-teamcity commented Jul 31, 2022

cockroach-teamcity commented Jul 31, 2022

cockroach-teamcity commented Aug 1, 2022

cockroach-teamcity commented Aug 2, 2022

cockroach-teamcity commented Aug 3, 2022

cockroach-teamcity commented Aug 3, 2022

cockroach-teamcity commented Aug 3, 2022

cockroach-teamcity commented Aug 4, 2022

rafiss commented Aug 4, 2022

cockroach-teamcity commented Aug 5, 2022

abarganier commented Aug 5, 2022

cockroach-teamcity commented Aug 6, 2022

abarganier commented Aug 6, 2022 • edited Loading

cockroach-teamcity commented Aug 7, 2022

cockroach-teamcity commented Aug 8, 2022

abarganier commented Aug 8, 2022

cockroach-teamcity commented Jul 28, 2022 •

edited by cockroach-jira-scripts

Loading

abarganier commented Aug 6, 2022 •

edited

Loading