backupccl: improve backup-restore roundtrip testing of multiple tenants

[msbutler]

A recent escalation revealed that the following queries from the system tenant would break RESTORE's of tenants:

CREATE TENANT foo
BACKUP  INTO 'nodelocal://1/clusterwide' with include_all_secondary_tenants;
CREATE TENANT baz
BACKUP  INTO LATEST IN 'nodelocal://1/clusterwide' with include_all_secondary_tenants;
RESTORE TENANT 2 FROM LATEST IN 'nodelocal://1/clusterwide';

We ought to develop a unit test which runs the following system tenant workload:

1. Runs various operations on application tenants (like create, drop, alter tenant, run c2c, upgrade tenant).
2. Runs incremental backups of individual tenants and with the include_all_virtual_clusters arg at various points in this workload
3. Restores these incremental backups

A poor man's version of this test is in #108743

Jira issue: CRDB-30619

[andy-kimball]

@stevendanna, is there any update on this? We'd like to re-enable incremental backups.

[stevendanna]

@andy-kimball Will have an update for you on this one by the end of the week.

[andy-kimball]

Following up here, any update?

[stevendanna]

Apologies that this has sat for so long. I've added the tests outline in this issue here:

#113851

I've run a stress run with these new nemeses in place for 2 days on a GCE worker. I still have this running and will keep an eye on it over this week. While simple, I've run it with past bugs re-inserted to ensure that this randomised approach would have caught the issue over time.

We've hit one issue so far:

• backupccl: failed to write metadata SST during TestTenantBackupNemesis #113852

This is in a default-off feature, but I am currently investigating to make sure that it isn't indicative of another problem.

Note that correct tenant restores currently still requires that tenant IDs are never re-used. We believe that this is the case in CC with the exception of manual action. That will be the case until we either move to per-tenant backup schedules or remove the use of ClearRange in the tenant GC path. We likely should do both over time.

I think we can re-enable incrementals for CC once we (1) merge and backport this test, and (2) I finish investigating the above failure. Both of which I expect to happen this week.

NB: We are currently investigating a newly issue discovered the the CCT cluster related to protected timestamps. While not strictly related to incremental tenant backups, depending on what we find, it may also have an impact for incremental tenant backups. However, the investigation there is going well so we should know more shortly.

NB2: The universe heard my optimism and immediately rewarded it with a test failure. I've updated this message to reflect that. I think this failure wasn't caught on my GCEWorker as I had disabled metamorphic testing when investigating another issue.