Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

release-22.1: roachprod: support --secure flag for start-tenant #85719

Merged
merged 2 commits into from
Aug 8, 2022

Conversation

abarganier
Copy link
Contributor

@abarganier abarganier commented Aug 7, 2022

Backport 1/1 commits from #82248.

/cc @cockroachdb/release


The in-development tenant streaming features require a secure tenant
to test them properly.

This change adds support for the --secure flag when starting a
tenant. When passed, we use the host cluster to generate tenant-client
certificates and distribute them to the tenant cluster, along with the
CA and previously created client certificates.

For simplicity, we've stuck with using a single CA for all certificate
creation.

Release note: None


Backport notes

This PR is a follow up to a previous backport (#84371) which introduced functionality to support tenant scoped client certificates & authz checks into CockroachDB v22.1.

While #84371 successfully provided all the necessary changes within CockroachDB to support these new certificates, it failed to include some necessary roachprod changes (#82248) required to make use of these new certificates in roachprod/roachtests, and more generally, to run multitenant roachtests in secure mode.

This led to roachtest failures on the release-22.1 branch (#85211). I've confirmed that the failing tests pass with the inclusion of #82248.

Furthermore, with support for secure multitenant roachtests, we no longer have to skip the acceptance/multitenant roachtest on the release-22.1 branch. This PR removes the Skip flag from the test registry for acceptance/multitenant.

Lastly, it adds the install.SecureOption(true) flag to the cluster settings used in the setup of the multitenant-upgrade and acceptance/multitenant roachtests.


Release justification: fix for a roachtest failure marked as a release blocked for the release-22.1 branch.

The in-development tenant streaming features require a secure tenant
to test them properly.

This change adds support for the --secure flag when starting a
tenant. When passed, we use the host cluster to generate tenant-client
certificates and distribute them to the tenant cluster, along with the
CA and previously created client certificates.

For simplicity, we've stuck with using a single CA for all certificate
creation.

Release note: None
@blathers-crl
Copy link

blathers-crl bot commented Aug 7, 2022

Thanks for opening a backport.

Please check the backport criteria before merging:

  • Patches should only be created for serious issues or test-only changes.
  • Patches should not break backwards-compatibility.
  • Patches should change as little code as possible.
  • Patches should not change on-disk formats or node communication protocols.
  • Patches should not add new functionality.
  • Patches must not add, edit, or otherwise modify cluster versions; or add version gates.
If some of the basic criteria cannot be satisfied, ensure that the exceptional criteria are satisfied within.
  • There is a high priority need for the functionality that cannot wait until the next release and is difficult to address in another way.
  • The new functionality is additive-only and only runs for clusters which have specifically “opted in” to it (e.g. by a cluster setting).
  • New code is protected by a conditional check that is trivial to verify and ensures that it only runs for opt-in clusters.
  • The PM and TL on the team that owns the changed code have signed off that the change obeys the above rules.

Add a brief release justification to the body of your PR to justify this backport.

Some other things to consider:

  • What did we do to ensure that a user that doesn’t know & care about this backport, has no idea that it happened?
  • Will this work in a cluster of mixed patch versions? Did we test that?
  • If a user upgrades a patch version, uses this feature, and then downgrades, what happens?

@cockroach-teamcity
Copy link
Member

This change is Reviewable

This PR updates the `acceptance/multitenant` and `multitenant-upgrade`
roachtests to run in secure mode, now that the necessary infra. to run
multitenant roachtests in secure mode has been added to roachprod.

It also updates the roachprod setup to detect whether or not the
cockroach binary supports tenant-scoped client certificates via the
`--help` command's output.

Release note: none
@abarganier abarganier marked this pull request as ready for review August 7, 2022 18:33
@abarganier abarganier requested a review from a team as a code owner August 7, 2022 18:33
@abarganier
Copy link
Contributor Author

Multitenant roachtest outputs with these changes:

acceptance/multitenant

23:18:55 test_runner.go:579: [w0] starting test: acceptance/multitenant:1
23:18:55 test_runner.go:745: [w0] === RUN   acceptance/multitenant
=== RUN   acceptance/multitenant
23:18:55 cluster.go:636: test status: uploading file
23:18:55 cluster_synced.go:1388: abarganier-1659741417-01-n4cpu4: putting (dist) /Users/abarganier/go/src/github.com/cockroachdb/cockroach/cockroach-linux-2.6.32-gnu-amd64 ./cockroach
23:19:16 cluster_synced.go:1586:
23:19:16 cluster_synced.go:1589:    1: done
23:19:16 cluster_synced.go:1589:    2: done
23:19:16 cluster_synced.go:1589:    3: done
23:19:16 cluster_synced.go:1589:    4: done
23:19:16 cluster.go:636: test status:
23:19:16 cluster.go:636: test status: starting nodes :1-4
abarganier-1659741417-01-n4cpu4: checking certs.tar
abarganier-1659741417-01-n4cpu4: initializing certs
abarganier-1659741417-01-n4cpu4: distributing certs
23:19:22 cockroach.go:174: abarganier-1659741417-01-n4cpu4: starting nodes
23:19:25 cockroach.go:211: abarganier-1659741417-01-n4cpu4: initializing cluster
23:19:26 cockroach.go:218: warning: --url specifies user/password, but command "init" does not accept user/password details - details ignored
Cluster successfully initialized
23:19:26 cockroach.go:225: abarganier-1659741417-01-n4cpu4: setting cluster settings
23:19:27 cockroach.go:231: SET CLUSTER SETTING
SET CLUSTER SETTING
SET CLUSTER SETTING
23:19:37 cluster.go:636: test status: getting ./certs
23:19:37 cluster_synced.go:1755: abarganier-1659741417-01-n4cpu4: getting (scp) ./certs /var/folders/d3/p3v925xd2v58pwy6njhnjx4h0000gq/T/roachtest-certs1076127082/certs
23:19:38 cluster.go:636: test status:
23:19:45 multitenant_utils.go:225: sql server for tenant 123 running at postgres://[email protected]:30258?sslcert=%2Fvar%2Ffolders%2Fd3%2Fp3v925xd2v58pwy6njhnjx4h0000gq%2FT%2Froachtest-certs1076127082%2Fcerts%2Fclient.root.crt&sslkey=%2Fvar%2Ffolders%2Fd3%2Fp3v925xd2v58pwy6njhnjx4h0000gq%2FT%2Froachtest-certs1076127082%2Fcerts%2Fclient.root.key&sslmode=verify-full&sslrootcert=%2Fvar%2Ffolders%2Fd3%2Fp3v925xd2v58pwy6njhnjx4h0000gq%2FT%2Froachtest-certs1076127082%2Fcerts%2Fca.crt
23:19:45 multitenant.go:44: test status: checking that a client can connect to the tenant server
23:19:45 multitenant.go:52: test status: stopping the server ahead of checking for the tenant server
23:19:47 multitenant_utils.go:146: mt cluster exited: <nil>
23:19:47 multitenant.go:57: test status: checking log file contents
23:19:50 test_runner.go:894: tearing down after success; see teardown.log
--- PASS: acceptance/multitenant (55.30s)
23:19:50 test_runner.go:636: [w0] test passed: acceptance/multitenant (run 1)

multitenant-upgrade

23:28:43 test_runner.go:579: [w0] starting test: multitenant-upgrade:1
23:28:43 test_runner.go:745: [w0] === RUN   multitenant-upgrade
=== RUN   multitenant-upgrade
23:28:43 cluster.go:636: test status: uploading file
23:28:43 cluster_synced.go:1388: abarganier-1659742011-01-n2cpu4: putting (dist) /Users/abarganier/go/src/github.com/cockroachdb/cockroach/cockroach-linux-2.6.32-gnu-amd64 ./cockroach
23:29:00 cluster_synced.go:1586:
23:29:00 cluster_synced.go:1589:    1: done
23:29:00 cluster_synced.go:1589:    2: done
23:29:00 cluster.go:636: test status:
23:29:02 cluster.go:636: test status: staging binary
23:29:02 staging.go:294: Resolved release url for cockroach version v21.2.14: https://s3.amazonaws.com/binaries.cockroachdb.com/cockroach-v21.2.14.linux-amd64.tgz
abarganier-1659742011-01-n2cpu4: staging cockroach release binary
   1:
   2:
23:29:05 cluster.go:636: test status:
23:29:05 cluster.go:636: test status: starting nodes :1
abarganier-1659742011-01-n2cpu4: checking certs.tar
abarganier-1659742011-01-n2cpu4: initializing certs
abarganier-1659742011-01-n2cpu4: distributing certs
23:29:11 cockroach.go:174: abarganier-1659742011-01-n2cpu4: starting nodes
23:29:14 cockroach.go:211: abarganier-1659742011-01-n2cpu4: initializing cluster
23:29:15 cockroach.go:218: warning: --url specifies user/password, but command "init" does not accept user/password details - details ignored
Cluster successfully initialized
23:29:15 cockroach.go:225: abarganier-1659742011-01-n2cpu4: setting cluster settings
23:29:16 cockroach.go:231: SET CLUSTER SETTING
SET CLUSTER SETTING
23:29:16 cluster.go:636: test status: getting ./certs
23:29:16 cluster_synced.go:1755: abarganier-1659742011-01-n2cpu4: getting (scp) ./certs /var/folders/d3/p3v925xd2v58pwy6njhnjx4h0000gq/T/roachtest-certs1141930400/certs
23:29:18 cluster.go:636: test status:
23:29:25 multitenant_utils.go:225: sql server for tenant 11 running at postgres://[email protected]:20011?sslcert=%2Fvar%2Ffolders%2Fd3%2Fp3v925xd2v58pwy6njhnjx4h0000gq%2FT%2Froachtest-certs1141930400%2Fcerts%2Fclient.root.crt&sslkey=%2Fvar%2Ffolders%2Fd3%2Fp3v925xd2v58pwy6njhnjx4h0000gq%2FT%2Froachtest-certs1141930400%2Fcerts%2Fclient.root.key&sslmode=verify-full&sslrootcert=%2Fvar%2Ffolders%2Fd3%2Fp3v925xd2v58pwy6njhnjx4h0000gq%2FT%2Froachtest-certs1141930400%2Fcerts%2Fca.crt
23:29:25 multitenant_upgrade.go:93: test status: checking that a client can connect to the tenant 11 server
23:29:25 multitenant_upgrade.go:105: test status: preserving downgrade option on host server
23:29:25 multitenant_upgrade.go:114: test status: upgrading host server
23:29:25 cluster.go:636: test status: stopping nodes :1
abarganier-1659742011-01-n2cpu4: stopping and waiting
23:29:27 cluster.go:636: test status: starting nodes :1
abarganier-1659742011-01-n2cpu4: checking certs.tar
23:29:28 cockroach.go:174: abarganier-1659742011-01-n2cpu4: starting nodes
23:29:32 cockroach.go:211: abarganier-1659742011-01-n2cpu4: initializing cluster
23:29:32 cockroach.go:225: abarganier-1659742011-01-n2cpu4: setting cluster settings
23:29:33 cluster.go:636: test status: getting ./certs
23:29:33 cluster_synced.go:1755: abarganier-1659742011-01-n2cpu4: getting (scp) ./certs /var/folders/d3/p3v925xd2v58pwy6njhnjx4h0000gq/T/roachtest-certs1493261635/certs
23:29:35 cluster.go:636: test status:
23:29:36 multitenant_upgrade.go:120: test status: checking the pre-upgrade sql server still works after the KV binary upgrade
23:29:36 multitenant_upgrade.go:126: test status: creating a new tenant 12
23:29:36 multitenant_upgrade.go:132: test status: starting tenant 12 server with older binary
23:29:42 multitenant_utils.go:225: sql server for tenant 12 running at postgres://[email protected]:20012?sslcert=%2Fvar%2Ffolders%2Fd3%2Fp3v925xd2v58pwy6njhnjx4h0000gq%2FT%2Froachtest-certs1493261635%2Fcerts%2Fclient.root.crt&sslkey=%2Fvar%2Ffolders%2Fd3%2Fp3v925xd2v58pwy6njhnjx4h0000gq%2FT%2Froachtest-certs1493261635%2Fcerts%2Fclient.root.key&sslmode=verify-full&sslrootcert=%2Fvar%2Ffolders%2Fd3%2Fp3v925xd2v58pwy6njhnjx4h0000gq%2FT%2Froachtest-certs1493261635%2Fcerts%2Fca.crt
23:29:42 multitenant_upgrade.go:137: test status: verifying that the tenant 12 server works and is at the earlier version
23:29:43 multitenant_upgrade.go:148: test status: creating a new tenant 13
23:29:43 multitenant_upgrade.go:154: test status: starting tenant 13 server with new binary
23:29:49 multitenant_utils.go:225: sql server for tenant 13 running at postgres://[email protected]:20013?sslcert=%2Fvar%2Ffolders%2Fd3%2Fp3v925xd2v58pwy6njhnjx4h0000gq%2FT%2Froachtest-certs1493261635%2Fcerts%2Fclient.root.crt&sslkey=%2Fvar%2Ffolders%2Fd3%2Fp3v925xd2v58pwy6njhnjx4h0000gq%2FT%2Froachtest-certs1493261635%2Fcerts%2Fclient.root.key&sslmode=verify-full&sslrootcert=%2Fvar%2Ffolders%2Fd3%2Fp3v925xd2v58pwy6njhnjx4h0000gq%2FT%2Froachtest-certs1493261635%2Fcerts%2Fca.crt
23:29:49 multitenant_upgrade.go:159: test status: verifying that the tenant 13 server works and is at the earlier version
23:29:50 multitenant_upgrade.go:170: test status: stopping the tenant 11 server ahead of upgrading
23:29:51 multitenant_utils.go:146: mt cluster exited: output in run_232923.736921000_n2_v21214cockroach_mt: v21.2.14/cockroach mt start-sql --certs-dir certs --tenant-id=11 --http-addr 0.0.0.0:8011 --kv-addrs 10.142.0.13:26257 --sql-addr 10.142.0.102:20011 --log="file-defaults: {dir: 'logs/mt-11', exit-on-error: false}" --store=cockroach-data-mt-11 returned: COMMAND_PROBLEM: exit status 1
23:29:51 multitenant_upgrade.go:173: test status: starting the tenant 11 server with the current binary
23:29:53 multitenant_utils.go:225: sql server for tenant 11 running at postgres://[email protected]:20011?sslcert=%2Fvar%2Ffolders%2Fd3%2Fp3v925xd2v58pwy6njhnjx4h0000gq%2FT%2Froachtest-certs1493261635%2Fcerts%2Fclient.root.crt&sslkey=%2Fvar%2Ffolders%2Fd3%2Fp3v925xd2v58pwy6njhnjx4h0000gq%2FT%2Froachtest-certs1493261635%2Fcerts%2Fclient.root.key&sslmode=verify-full&sslrootcert=%2Fvar%2Ffolders%2Fd3%2Fp3v925xd2v58pwy6njhnjx4h0000gq%2FT%2Froachtest-certs1493261635%2Fcerts%2Fca.crt
23:29:53 multitenant_upgrade.go:176: test status: verify tenant 11 server works with the new binary
23:29:53 multitenant_upgrade.go:187: test status: migrating the tenant 11 to the current version before kv is finalized
23:34:42 multitenant_upgrade.go:199: test status: finalizing the kv server
I220805 23:34:46.240058 583 testutils/sqlutils/sql_runner.go:235  [-] 1  SucceedsSoon: query 'SELECT version = crdb_internal.node_executable_version() FROM [SHOW CLUSTER SETTING version]': expected:
I220805 23:34:46.240058 583 testutils/sqlutils/sql_runner.go:235  [-] 1 +true
I220805 23:34:46.240058 583 testutils/sqlutils/sql_runner.go:235  [-] 1 +got:
I220805 23:34:46.240058 583 testutils/sqlutils/sql_runner.go:235  [-] 1 +false
I220805 23:34:47.284985 583 testutils/sqlutils/sql_runner.go:235  [-] 2  SucceedsSoon: query 'SELECT version = crdb_internal.node_executable_version() FROM [SHOW CLUSTER SETTING version]': expected:
I220805 23:34:47.284985 583 testutils/sqlutils/sql_runner.go:235  [-] 2 +true
I220805 23:34:47.284985 583 testutils/sqlutils/sql_runner.go:235  [-] 2 +got:
I220805 23:34:47.284985 583 testutils/sqlutils/sql_runner.go:235  [-] 2 +false
I220805 23:34:48.326903 583 testutils/sqlutils/sql_runner.go:235  [-] 3  SucceedsSoon: query 'SELECT version = crdb_internal.node_executable_version() FROM [SHOW CLUSTER SETTING version]': expected:
I220805 23:34:48.326903 583 testutils/sqlutils/sql_runner.go:235  [-] 3 +true
I220805 23:34:48.326903 583 testutils/sqlutils/sql_runner.go:235  [-] 3 +got:
I220805 23:34:48.326903 583 testutils/sqlutils/sql_runner.go:235  [-] 3 +false
I220805 23:34:49.369893 583 testutils/sqlutils/sql_runner.go:235  [-] 4  SucceedsSoon: query 'SELECT version = crdb_internal.node_executable_version() FROM [SHOW CLUSTER SETTING version]': expected:
I220805 23:34:49.369893 583 testutils/sqlutils/sql_runner.go:235  [-] 4 +true
I220805 23:34:49.369893 583 testutils/sqlutils/sql_runner.go:235  [-] 4 +got:
I220805 23:34:49.369893 583 testutils/sqlutils/sql_runner.go:235  [-] 4 +false
I220805 23:34:50.415083 583 testutils/sqlutils/sql_runner.go:235  [-] 5  SucceedsSoon: query 'SELECT version = crdb_internal.node_executable_version() FROM [SHOW CLUSTER SETTING version]': expected:
I220805 23:34:50.415083 583 testutils/sqlutils/sql_runner.go:235  [-] 5 +true
I220805 23:34:50.415083 583 testutils/sqlutils/sql_runner.go:235  [-] 5 +got:
I220805 23:34:50.415083 583 testutils/sqlutils/sql_runner.go:235  [-] 5 +false
I220805 23:34:51.458200 583 testutils/sqlutils/sql_runner.go:235  [-] 6  SucceedsSoon: query 'SELECT version = crdb_internal.node_executable_version() FROM [SHOW CLUSTER SETTING version]': expected:
I220805 23:34:51.458200 583 testutils/sqlutils/sql_runner.go:235  [-] 6 +true
I220805 23:34:51.458200 583 testutils/sqlutils/sql_runner.go:235  [-] 6 +got:
I220805 23:34:51.458200 583 testutils/sqlutils/sql_runner.go:235  [-] 6 +false
I220805 23:34:52.504847 583 testutils/sqlutils/sql_runner.go:235  [-] 7  SucceedsSoon: query 'SELECT version = crdb_internal.node_executable_version() FROM [SHOW CLUSTER SETTING version]': expected:
I220805 23:34:52.504847 583 testutils/sqlutils/sql_runner.go:235  [-] 7 +true
I220805 23:34:52.504847 583 testutils/sqlutils/sql_runner.go:235  [-] 7 +got:
I220805 23:34:52.504847 583 testutils/sqlutils/sql_runner.go:235  [-] 7 +false
I220805 23:34:53.543526 583 testutils/sqlutils/sql_runner.go:235  [-] 8  SucceedsSoon: query 'SELECT version = crdb_internal.node_executable_version() FROM [SHOW CLUSTER SETTING version]': expected:
I220805 23:34:53.543526 583 testutils/sqlutils/sql_runner.go:235  [-] 8 +true
I220805 23:34:53.543526 583 testutils/sqlutils/sql_runner.go:235  [-] 8 +got:
I220805 23:34:53.543526 583 testutils/sqlutils/sql_runner.go:235  [-] 8 +false
I220805 23:34:54.580942 583 testutils/sqlutils/sql_runner.go:235  [-] 9  SucceedsSoon: query 'SELECT version = crdb_internal.node_executable_version() FROM [SHOW CLUSTER SETTING version]': expected:
I220805 23:34:54.580942 583 testutils/sqlutils/sql_runner.go:235  [-] 9 +true
I220805 23:34:54.580942 583 testutils/sqlutils/sql_runner.go:235  [-] 9 +got:
I220805 23:34:54.580942 583 testutils/sqlutils/sql_runner.go:235  [-] 9 +false
I220805 23:34:55.623664 583 testutils/sqlutils/sql_runner.go:235  [-] 10  SucceedsSoon: query 'SELECT version = crdb_internal.node_executable_version() FROM [SHOW CLUSTER SETTING version]': expected:
I220805 23:34:55.623664 583 testutils/sqlutils/sql_runner.go:235  [-] 10 +true
I220805 23:34:55.623664 583 testutils/sqlutils/sql_runner.go:235  [-] 10 +got:
I220805 23:34:55.623664 583 testutils/sqlutils/sql_runner.go:235  [-] 10 +false
I220805 23:34:56.660875 583 testutils/sqlutils/sql_runner.go:235  [-] 11  SucceedsSoon: query 'SELECT version = crdb_internal.node_executable_version() FROM [SHOW CLUSTER SETTING version]': expected:
I220805 23:34:56.660875 583 testutils/sqlutils/sql_runner.go:235  [-] 11 +true
I220805 23:34:56.660875 583 testutils/sqlutils/sql_runner.go:235  [-] 11 +got:
I220805 23:34:56.660875 583 testutils/sqlutils/sql_runner.go:235  [-] 11 +false
I220805 23:34:57.703998 583 testutils/sqlutils/sql_runner.go:235  [-] 12  SucceedsSoon: query 'SELECT version = crdb_internal.node_executable_version() FROM [SHOW CLUSTER SETTING version]': expected:
I220805 23:34:57.703998 583 testutils/sqlutils/sql_runner.go:235  [-] 12 +true
I220805 23:34:57.703998 583 testutils/sqlutils/sql_runner.go:235  [-] 12 +got:
I220805 23:34:57.703998 583 testutils/sqlutils/sql_runner.go:235  [-] 12 +false
I220805 23:34:58.749996 583 testutils/sqlutils/sql_runner.go:235  [-] 13  SucceedsSoon: query 'SELECT version = crdb_internal.node_executable_version() FROM [SHOW CLUSTER SETTING version]': expected:
I220805 23:34:58.749996 583 testutils/sqlutils/sql_runner.go:235  [-] 13 +true
I220805 23:34:58.749996 583 testutils/sqlutils/sql_runner.go:235  [-] 13 +got:
I220805 23:34:58.749996 583 testutils/sqlutils/sql_runner.go:235  [-] 13 +false
I220805 23:34:59.788337 583 testutils/sqlutils/sql_runner.go:235  [-] 14  SucceedsSoon: query 'SELECT version = crdb_internal.node_executable_version() FROM [SHOW CLUSTER SETTING version]': expected:
I220805 23:34:59.788337 583 testutils/sqlutils/sql_runner.go:235  [-] 14 +true
I220805 23:34:59.788337 583 testutils/sqlutils/sql_runner.go:235  [-] 14 +got:
I220805 23:34:59.788337 583 testutils/sqlutils/sql_runner.go:235  [-] 14 +false
23:35:00 multitenant_upgrade.go:205: test status: stopping the tenant 12 server ahead of upgrading
23:35:01 multitenant_utils.go:146: mt cluster exited: output in run_232941.513047000_n2_v21214cockroach_mt: v21.2.14/cockroach mt start-sql --certs-dir certs --tenant-id=12 --http-addr 0.0.0.0:8012 --kv-addrs 10.142.0.13:26257 --sql-addr 10.142.0.102:20012 --log="file-defaults: {dir: 'logs/mt-12', exit-on-error: false}" --store=cockroach-data-mt-12 returned: COMMAND_PROBLEM: exit status 1
23:35:01 multitenant_upgrade.go:208: test status: starting the tenant 12 server with the current binary
23:35:03 multitenant_utils.go:225: sql server for tenant 12 running at postgres://[email protected]:20012?sslcert=%2Fvar%2Ffolders%2Fd3%2Fp3v925xd2v58pwy6njhnjx4h0000gq%2FT%2Froachtest-certs1493261635%2Fcerts%2Fclient.root.crt&sslkey=%2Fvar%2Ffolders%2Fd3%2Fp3v925xd2v58pwy6njhnjx4h0000gq%2FT%2Froachtest-certs1493261635%2Fcerts%2Fclient.root.key&sslmode=verify-full&sslrootcert=%2Fvar%2Ffolders%2Fd3%2Fp3v925xd2v58pwy6njhnjx4h0000gq%2FT%2Froachtest-certs1493261635%2Fcerts%2Fca.crt
23:35:03 multitenant_upgrade.go:211: test status: verify tenant 12 server works with the new binary
23:35:04 multitenant_upgrade.go:219: test status: migrating tenant 12 to the current version
23:39:52 multitenant_upgrade.go:225: test status: restarting the tenant 12 server to check it works after a restart
23:39:54 multitenant_utils.go:146: mt cluster exited: <nil>
23:39:56 multitenant_utils.go:225: sql server for tenant 12 running at postgres://[email protected]:20012?sslcert=%2Fvar%2Ffolders%2Fd3%2Fp3v925xd2v58pwy6njhnjx4h0000gq%2FT%2Froachtest-certs1493261635%2Fcerts%2Fclient.root.crt&sslkey=%2Fvar%2Ffolders%2Fd3%2Fp3v925xd2v58pwy6njhnjx4h0000gq%2FT%2Froachtest-certs1493261635%2Fcerts%2Fclient.root.key&sslmode=verify-full&sslrootcert=%2Fvar%2Ffolders%2Fd3%2Fp3v925xd2v58pwy6njhnjx4h0000gq%2FT%2Froachtest-certs1493261635%2Fcerts%2Fca.crt
23:39:56 multitenant_upgrade.go:229: test status: verify tenant 12 server works with the new binary after restart
23:39:56 multitenant_upgrade.go:237: test status: migrating tenant 13 to the current version
23:39:57 multitenant_upgrade.go:247: test status: restarting the tenant 13 server to check it works after a restart
23:39:59 multitenant_utils.go:146: mt cluster exited: <nil>
23:40:01 multitenant_utils.go:225: sql server for tenant 13 running at postgres://[email protected]:20013?sslcert=%2Fvar%2Ffolders%2Fd3%2Fp3v925xd2v58pwy6njhnjx4h0000gq%2FT%2Froachtest-certs1493261635%2Fcerts%2Fclient.root.crt&sslkey=%2Fvar%2Ffolders%2Fd3%2Fp3v925xd2v58pwy6njhnjx4h0000gq%2FT%2Froachtest-certs1493261635%2Fcerts%2Fclient.root.key&sslmode=verify-full&sslrootcert=%2Fvar%2Ffolders%2Fd3%2Fp3v925xd2v58pwy6njhnjx4h0000gq%2FT%2Froachtest-certs1493261635%2Fcerts%2Fca.crt
23:40:01 multitenant_upgrade.go:251: test status: verify tenant 13 server works with the new binary after restart
23:40:02 multitenant_upgrade.go:258: test status: creating tenant 14 at the new version
23:40:02 multitenant_upgrade.go:264: test status: verifying the tenant 14 works and has the proper version
23:40:08 multitenant_utils.go:225: sql server for tenant 14 running at postgres://[email protected]:20014?sslcert=%2Fvar%2Ffolders%2Fd3%2Fp3v925xd2v58pwy6njhnjx4h0000gq%2FT%2Froachtest-certs1493261635%2Fcerts%2Fclient.root.crt&sslkey=%2Fvar%2Ffolders%2Fd3%2Fp3v925xd2v58pwy6njhnjx4h0000gq%2FT%2Froachtest-certs1493261635%2Fcerts%2Fclient.root.key&sslmode=verify-full&sslrootcert=%2Fvar%2Ffolders%2Fd3%2Fp3v925xd2v58pwy6njhnjx4h0000gq%2FT%2Froachtest-certs1493261635%2Fcerts%2Fca.crt
23:40:08 multitenant_upgrade.go:276: test status: restarting the tenant 14 server to check it works after a restart
23:40:10 multitenant_utils.go:146: mt cluster exited: <nil>
23:40:12 multitenant_utils.go:225: sql server for tenant 13 running at postgres://[email protected]:20013?sslcert=%2Fvar%2Ffolders%2Fd3%2Fp3v925xd2v58pwy6njhnjx4h0000gq%2FT%2Froachtest-certs1493261635%2Fcerts%2Fclient.root.crt&sslkey=%2Fvar%2Ffolders%2Fd3%2Fp3v925xd2v58pwy6njhnjx4h0000gq%2FT%2Froachtest-certs1493261635%2Fcerts%2Fclient.root.key&sslmode=verify-full&sslrootcert=%2Fvar%2Ffolders%2Fd3%2Fp3v925xd2v58pwy6njhnjx4h0000gq%2FT%2Froachtest-certs1493261635%2Fcerts%2Fca.crt
23:40:12 multitenant_upgrade.go:280: test status: verifying the post-upgrade tenant works and has the proper version
23:40:14 multitenant_utils.go:146: mt cluster exited: <nil>
23:40:15 multitenant_utils.go:146: mt cluster exited: <nil>
23:40:16 multitenant_utils.go:146: mt cluster exited: <nil>
23:40:18 multitenant_utils.go:146: mt cluster exited: <nil>
23:40:18 test_runner.go:894: tearing down after success; see teardown.log
--- PASS: multitenant-upgrade (695.33s)
23:40:18 test_runner.go:636: [w0] test passed: multitenant-upgrade (run 1)
23:40:19 test_runner.go:698: [w0] no perf artifacts exist on node :2
23:40:19 test_runner.go:698: [w0] no perf artifacts exist on node :1
23:40:19 work_pool.go:105: [w0] No tests that can reuse cluster abarganier-1659742011-01-n2cpu4 [tag:] (2 nodes) found (or there are no further tests to run). Destroying.
23:40:19 cluster.go:1559: [w0] destroying cluster abarganier-1659742011-01-n2cpu4 [tag:] (2 nodes)...
23:40:22 roachprod.go:1126: Destroying cluster abarganier-1659742011-01-n2cpu4 with 2 nodes
23:40:44 roachprod.go:1117: OK
23:40:44 cluster.go:1567: [w0] destroying cluster abarganier-1659742011-01-n2cpu4 [tag:] (2 nodes)... done
23:40:44 test_runner.go:499: [w0] no work remaining; runWorker is bailing out...
no work remaining; runWorker is bailing out...
23:40:44 test_runner.go:451: [w0] Worker exiting; destroying cluster.
23:40:44 test_runner.go:303: PASS
PASS

Copy link
Collaborator

@rafiss rafiss left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ty!

@abarganier
Copy link
Contributor Author

TFTR!

@abarganier abarganier merged commit a59c4dc into cockroachdb:release-22.1 Aug 8, 2022
Copy link
Collaborator

@stevendanna stevendanna left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Belatedly also giving a +1 here. Thanks for backporting this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants