roachtest: add ability to perform multiple upgrades in mixedversion

[renatolabs]

This commit adds the ability for mixedversion tests to perform
multiple upgrades in certain tests runs. It has been observed that
certain types of errors only manifest when features are enabled in
older releases, and this makes it possible for us to get some coverage
for these scenarios.

By default, mixedversion tests will perform a random number of
upgrades (up to a maximum of 3). Test authors are able to override
this default by providing a minimum, maximum, or exact number of
upgrades test runs should go through.

Epic: CRDB-19321

Release note: None

[cockroach-teamcity]

This change is 

[renatolabs]

There's a likely legit bug that causes backup-restore/mixed-version to fail on this branch when a RESTORE is attempted while the cluster is finalizing the 22.1 -> 22.2 upgrade:

[mixed-version-test/20_run-verify-some-backups] 01:51:05 runner.go:336: mixed-version test failure while running step 20 (run "verify some backups"): mixed-version: waiting for job to finish: job 891269071918759937 failed with error: version mismatch for descriptor 132, expected version 2, got 1

I'll let @cockroachdb/disaster-recovery deal with this failure as they see fit when it happens in the nightly build.

[smg260]

The feature generally makes sense to me.

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @herkolategan, @renatolabs, and @srosenberg)

---

pkg/cmd/roachtest/roachtestutil/mixedversion/mixedversion.go line 929 at r1 (raw file):

	if test.options.minUpgrades > test.options.maxUpgrades {
		err := fmt.Errorf(
			"invalid test options: minUpgrades=%d maxUpgrades=%d",

minor: might make sense to be explicit about the error here.

e.g. maxUpgrades (%d) must be greater than minUpgrades (%d)

[renatolabs]

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @herkolategan, @smg260, and @srosenberg)

---

pkg/cmd/roachtest/roachtestutil/mixedversion/mixedversion.go line 929 at r1 (raw file):

Previously, smg260 (Miral Gadani) wrote…

minor: might make sense to be explicit about the error here.

e.g. maxUpgrades (%d) must be greater than minUpgrades (%d)

Makes sense, done.

[renatolabs]

Added another commit to adapt the new cdc/mixed-versions to multiple upgrades (also updated test ownership back to CDC; always happy to help debug mixed-version issues, but it makes more sense for cdc to own that test).

@jayshrivastava FYI.

[renatolabs]

TFTR!

bors r=smg260

[craig]

Build succeeded:

• Bazel Essential CI (Cockroach)

[blathers-crl]

Encountered an error creating backports. Some common things that can go wrong:

1. The backport branch might have already existed.
2. There was a merge conflict.
3. The backport branch contained merge commits.

You might need to create your backport manually using the backport tool.

---

error creating merge commit from 2b00746 to blathers/backport-release-23.1-108773: POST https://api.github.com/repos/cockroachdb/cockroach/merges: 409 Merge conflict []

you may need to manually resolve merge conflicts with the backport tool.

Backport to branch 23.1.x failed. See errors above.

---

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}