Skip to content

Commit

Permalink
roachtest: update mixed-version backup to use new framework
Browse files Browse the repository at this point in the history
This updates the `backup/mixed-version` roachtest to use the recently
introduced mixed-version roachtest framework (`mixedversion` package).

The main behavior exercised remains the same: backups are taken in
mixed-binary state, and those backups are restored and verified at the
end of the test. However, this commit also improves the coverage of
mixed-version backup testing in a few ways:

* **Randomization**. By virtue of using the new framework, most runs
will be different from one another since the order of actions taken by
the test will be different. Previously, backups would always be taken
with 2 nodes in the old version and 2 nodes in the new version. Now,
backups can be taken when an arbitrary number of nodes is running the
new version. As a consequence, it's also possible that some executions
will attempt backups when all nodes are running a new binary version,
but the cluster version itself has not been updated. Other points of
new randomization include the choice of the node's external dir where
backups are stored, which node to connect to when running certain
statements, and how much to wait between backups.

* **Backup Options**. Backups will randomly be created with
`revision_history` enabled, or with an `encryption_passphrase`.

* **Downgrades**. The cluster is also downgraded in mixed-version
tests. No downgrades happened in that test before this commit.

* **Workload**. Instead of using fixed call to `generate_series` to
generate data between backups, the test now runs the `bank` workload
continuously during the test. A random wait between backups allows the
workload to make changes to the underlying table during the test and
for the backups to be taken while writes are taking place.

* **Finalization**: the test _may_ attempt to create a backup as the
upgrade is finalizing (i.e., migrations are running and cluster
version is advancing).

In addition, this test will also see improved coverage as we make more
improvements to test plans generated by the `mixedversion` package.
These changes will create more backup scenarios in the future without
requiring any code changes to this test.

This test has already helped us uncover one backup bug (#97953).

Epic: CRDB-19321

Release note: None
  • Loading branch information
renatolabs committed Mar 10, 2023
1 parent 9b202ba commit 0b7da89
Show file tree
Hide file tree
Showing 4 changed files with 847 additions and 333 deletions.
1 change: 1 addition & 0 deletions pkg/cmd/roachtest/tests/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,7 @@ go_library(
"liquibase_blocklist.go",
"loss_of_quorum_recovery.go",
"many_splits.go",
"mixed_version_backup.go",
"mixed_version_cdc.go",
"mixed_version_change_replicas.go",
"mixed_version_decl_schemachange_compat.go",
Expand Down
Loading

0 comments on commit 0b7da89

Please sign in to comment.