backupccl: teach makeSimpleImportSpans to exclude backup data on spans that were later reintroduced

[msbutler]

Currently RESTORE cannot properly handle restoring a table that had a non-mvcc rollback, without adding some logic to makeSimpleImportSpans.

Consider the following timeline:

1. Begin IMPORT INTO on existing table foo, ingest some data
2. Backup foo
3. Rollback IMPORT via clearRange
4. Incremental backup of foo, with a full reintroduction of foo’s span
5. RESTORE foo: the imported data leaks into the restore

• See test here. grep for “this is bad”

This occurs because 1) makeSimpleImportSpans naively includes the backup file with the imported data as part of foo's restoreSpanEntry and 2) the backups have MVCC delete history related to the rollback. This was never a problem in previous releases because offline importing data were not backed up until the import succeeded.

To address this, makeSimpleImportSpan needs to exclude backup files for a span that was later reintroduced due to a non-mvcc bulk op.

Jira issue: CRDB-19262

[blathers-crl]

cc @cockroachdb/bulk-io

[adityamaru]

Is this a release blocker on 22.1? That seems like a mistake, should it be 22.2 instead?

[msbutler]

Well, it turns out this bug is in 22.1, and most likely in earlier releases, but only for cluster backup chains, as they have been backing up offline tables for as long they have been around, I think .

I confirmed that this bug is on 22.1 by executing the sql commands in my data driven test on a 22.1 roachprod cluster. See my sql shell log with the commands I ran. The bug manifests in the following ways:

• If table foo is empty before the cancelled import begins, the cancelled imported data appears on the restored cluster/database/table
• If table foofoo is non-empty, foofoo does not seem to be available on the restored cluster, even if the import was cancelled before the incremental backup was taken. I'm not sure why yet.
• This behavior is consistent regardless of value of bulkio.import_at_current_time.enabled

I have not confirmed this bug is on 21.2 because it's harder to reproduce without the jobs.debug.pausepoints cluster setting. But the source code indicates that cluster backups will back up offline data, so I bet this bug is on there too.

[msbutler]

verified this bug is on 21.2.15 using the following cmds on a roachprod cluster. Note that descriptor resolution for cluster backups do not filter offline tables (just dropped ones), likely for job resolution on the restored cluster.

use bank;
CREATE TABLE bank2 (id INT8 NOT NULL, balance INT8 NULL, payload STRING NULL);
EXPORT INTO CSV 'gs://dt-backup-tests/michael-rangkey?AUTH=implicit' FROM TABLE bank; 
IMPORT INTO bank2 (id, balance,payload) CSV DATA ('gs://dt-backup-tests/michael-rangkey/export*-n*.0.csv?AUTH=implicit') WITH detached;

# immediately pause the import
PAUSE JOB [import_job]

# Run the full backup
BACKUP INTO 'gs://dt-backup-tests/michael-rangkey?AUTH=implicit';

# cancel the import 
PAUSE JOB [import_job]

# Run an incremental backup
BACKUP INTO LATEST IN 'gs://dt-backup-tests/michael-rangkey?AUTH=implicit';

USE defaultdb;
DROP bank;

# Restore to new database 
RESTORE DATABASE bank FROM LATEST IN 'gs://dt-backup-tests/michael-rangkey?AUTH=implicit';

# See the rolled back imported data
SELECT count(*) FROM bank.bank2;

[msbutler]

on 22.1, I found a slightly different bug which i'll file an issue for once i get to the root cause.

On #87873, I tried to fix the cluster backup bug described above on 22.1 by excluding offline tables. For backups without revision history this works, but for backups with revision history (cluster/database backups), I found another surprise: the first incremental backup which observes the table back online will reintroduce the clear ranged data and restore will restore the clear ranged data :(. This behavior isn't apparent on master. I have yet to find a root cause, but I suspect the bug is in kv server.

[msbutler]

fixed by
master: #87312
22.2: #89102
22.1: #88488
21.2: #89019