Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

backupccl: revision_history full backups can fail when attempting to merge spans beyond table GC TTL #66797

Closed
pbardea opened this issue Jun 23, 2021 · 0 comments · Fixed by #66840
Assignees
Labels
A-disaster-recovery C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. T-disaster-recovery

Comments

@pbardea
Copy link
Contributor

pbardea commented Jun 23, 2021

Describe the problem

During backup planning, backup attempts to find index spans that can be merged to reduce the number of individual spans that needed their timestamp protected. One of the checks performed is a KV scan between non-contiguous public indexes to check that there is no data in the dropped index. We check this for every version of the schema for backups with revision_history.

When considering a version of the schema that is older than the GC TTL of the table, this scan is illegal. This raises an error in backup. As an example, consider the following scenario where x represents the existence of a public index at that time:

              |
          t@1 |     xxxxxxxx|xxxxxxxxxxxxxxxxxxxxxxx
          t@2 |     xxxxxxxx|xxxxxxxxx
          t@3 |             |
          t@4 |     xxxxxxxx|xxxxxxxxx
              ----------------------------------------
                    t1    gc_tresh    t2            t3

Backup will first look at the schema at t3, see only 1 index (t@1) and continue. Then it will look at t1 and sees 2 new indexes that it can potentially merge, t@2 and t@4. It scans at t@3 at timestamp t1 which will fail the backup since the scan is before the gc threshold.

One possible solution to this problem would be to treat reads that return the "exceed gc threshold" error as *non-*empty (and we don't merge).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-disaster-recovery C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. T-disaster-recovery
Projects
No open projects
Archived in project
Development

Successfully merging a pull request may close this issue.

1 participant