Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

backupccl: add option to validate all files in a backup via SHOW BACKUP #77694

Closed
msbutler opened this issue Mar 11, 2022 · 3 comments · Fixed by #80491
Closed

backupccl: add option to validate all files in a backup via SHOW BACKUP #77694

msbutler opened this issue Mar 11, 2022 · 3 comments · Fixed by #80491
Assignees
@msbutler
Labels
A-disaster-recovery branch-release-22.1 Used to mark GA and release blockers, technical advisories, and bugs for 22.1 C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) T-disaster-recovery

Comments

@msbutler
Copy link
Collaborator

msbutler commented Mar 11, 2022 •
edited by cockroach-jira-scripts
Loading

SHOW BACKUP only shows the user data from the backup manifest. There should be an option to SHOW BACKUPS which checks that each file in a backup is in its expected place.

Jira issue: CRDB-13718
@msbutler msbutler added C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) A-disaster-recovery T-disaster-recovery labels Mar 11, 2022
@msbutler msbutler self-assigned this Mar 11, 2022
@blathers-crl
Copy link

blathers-crl bot commented Mar 11, 2022

cc @cockroachdb/bulk-io

@msbutler msbutler mentioned this issue Mar 22, 2022
Merged
msbutler added a commit to msbutler/cockroach that referenced this issue Mar 23, 2022
@msbutler
backupccl: SHOW BACKUP FILES IN (on collection) returns the full SST …
72c970d 
…path

Previously, SHOW BACKUP FILES on a backup collection  would return the  SST
file path relative to the manifest directory. Given that the incremental backup
and full backup manifests are stored in different directories, the file paths
that SHOW BACKUP FILES should reflect that.

This patch changes the path `SHOW BACKUP FILES IN` returns to the backup
path relative to the collection root. As an example:

Previously, the command `SHOW BACKUP FILES LATEST IN s3://mybackups`, would
return:

data/001.SST // from a full backup
data/002.SST  // from an incremental backup

Now, the command will return (assuming the full and inc live in same subdir):

/2020/12/25-060000.00/data/001.SST
/2020/12/25-060000.00/20201225/070000.00/data/002.SST

Note: when a user passes the incremental_location parameter, the output result
will be slightly misleading because the incrementals will have a different
collection root. To aid in this confusion, I added a backup_type column
equal to 'incremental' or 'full'.

I plan to test this change in the PR for cockroachdb#77694

Release note: None
craig bot pushed a commit that referenced this issue Mar 24, 2022
@msbutler
Merge #78251
d51cd4f 
78251: backupccl: SHOW BACKUP FILES IN (on collection) returns the full SST path r=dt a=msbutler

 backupccl: SHOW BACKUP FILES IN (on collection) returns the full SST path
    
Previously, SHOW BACKUP FILES on a backup collection  would return the  SST
file path relative to the manifest directory. Given that the incremental backup
and full backup manifests are stored in different directories, the file paths
that SHOW BACKUP FILES should reflect that.

This patch changes the path `SHOW BACKUP FILES IN` returns to the backup
path relative to the collection root. As an example:

Previously, the command `SHOW BACKUP FILES LATEST IN s3://mybackups`, would
return:
```
data/001.SST // from a full backup
data/002.SST  // from an incremental backup
```
Now, the command will return (assuming the full and inc live in same subdir):
```
/2020/12/25-060000.00/data/001.SST
/2020/12/25-060000.00/20201225/070000.00/data/002.SST
```
Note: when a user passes the incremental_location parameter, the output result
will be slightly misleading because the incrementals will have a different
collection root. To aid in this confusion, I added a backup_type column
equal to 'incremental' or 'full'.

I plan to test this change in the PR for #77694

Release note: None

Co-authored-by: Michael Butler <[email protected]>
msirek pushed a commit to msirek/cockroach that referenced this issue Mar 24, 2022
@msbutler
backupccl: SHOW BACKUP FILES IN (on collection) returns the full SST …
30587ed 
…path

Previously, SHOW BACKUP FILES on a backup collection  would return the  SST
file path relative to the manifest directory. Given that the incremental backup
and full backup manifests are stored in different directories, the file paths
that SHOW BACKUP FILES should reflect that.

This patch changes the path `SHOW BACKUP FILES IN` returns to the backup
path relative to the collection root. As an example:

Previously, the command `SHOW BACKUP FILES LATEST IN s3://mybackups`, would
return:

data/001.SST // from a full backup
data/002.SST  // from an incremental backup

Now, the command will return (assuming the full and inc live in same subdir):

/2020/12/25-060000.00/data/001.SST
/2020/12/25-060000.00/20201225/070000.00/data/002.SST

Note: when a user passes the incremental_location parameter, the output result
will be slightly misleading because the incrementals will have a different
collection root. To aid in this confusion, I added a backup_type column
equal to 'incremental' or 'full'.

I plan to test this change in the PR for cockroachdb#77694

Release note: None
@blathers-crl
Copy link

blathers-crl bot commented Mar 28, 2022

Hi @msbutler, please add branch-* labels to identify which branch(es) this release-blocker affects.

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan.

@msbutler msbutler added the branch-release-22.1 Used to mark GA and release blockers, technical advisories, and bugs for 22.1 label Mar 28, 2022
@msbutler msbutler mentioned this issue Mar 28, 2022
Closed
@msbutler msbutler mentioned this issue Mar 31, 2022
Merged
msbutler added a commit to msbutler/cockroach that referenced this issue Apr 4, 2022
@msbutler
backupccl: allow user to run SHOW BACKUP on locality aware backups
827a9ec 
Previously, the user could only run  SHOW BACKUP on the default locality of a
backup. This patch allows users to pass in all backup localities, allowing
future changes to SHOW BACKUP (e.g. cockroachdb#77694) to analyze data from backup files
in non default locaties.

Fixes cockroachdb#78632

Release note (sql change): users can pass locality aware backup URIS to SHOW
BACKUP. This change only affects SHOW BACKUP with the new syntax: e.g.:

SHOW BACKUP FROM LATEST IN (uri1, uri2)

Currently, SHOW BACKUP doesn't return anything different for locality aware
backups, but it might in the future!
msbutler added a commit to msbutler/cockroach that referenced this issue Apr 6, 2022
@msbutler
backupccl: allow user to run SHOW BACKUP on locality aware backups
6292014 
Previously, the user could only run  SHOW BACKUP on the default locality of a
backup. This patch allows users to pass in all backup localities, allowing
future changes to SHOW BACKUP (e.g. cockroachdb#77694) to analyze data from backup files
in non default locaties.

Users can not yet run SHOW BACKUP for locality aware backups created using the
incremental_location parameter.

Informs cockroachdb#78632

Release note (sql change): users can pass locality aware backup URIs to SHOW
BACKUP. This change only affects SHOW BACKUP with the new syntax: e.g.:

SHOW BACKUP FROM LATEST IN (<collectionURI>, <localityURI1>, <localityURI2>)

Users can not yet run SHOW BACKUP for locality aware backups created using the
incremental_location parameter. I'm hoping to slip this into the 22.1 release!
msbutler added a commit to msbutler/cockroach that referenced this issue Apr 6, 2022
@msbutler
backupccl: allow user to run SHOW BACKUP on locality aware backups
d918e95 
Previously, the user could only run  SHOW BACKUP on the default locality of a
backup. This patch allows users to pass in all backup localities, allowing
future changes to SHOW BACKUP (e.g. cockroachdb#77694) to analyze data from backup files
in non default locaties.

Users can not yet run SHOW BACKUP for locality aware backups created using the
incremental_location parameter.

Informs cockroachdb#78632

Release note (sql change): users can pass locality aware backup URIs to SHOW
BACKUP. This change only affects SHOW BACKUP with the new syntax: e.g.:

SHOW BACKUP FROM LATEST IN (<collectionURI>, <localityURI1>, <localityURI2>)

Users can not yet run SHOW BACKUP for locality aware backups created using the
incremental_location parameter. I'm hoping to slip this into the 22.1 release!
msbutler added a commit to msbutler/cockroach that referenced this issue Apr 13, 2022
@msbutler
backupccl: allow user to run SHOW BACKUP on locality aware backups
4a93dbf 
Previously, the user could only run  SHOW BACKUP on the default locality of a
backup. This patch allows users to pass in all backup localities, allowing
future changes to SHOW BACKUP (e.g. cockroachdb#77694) to analyze data from backup files
in non default locaties.

Users can not yet run SHOW BACKUP for locality aware backups created using the
incremental_location parameter.

Informs cockroachdb#78632

Release note (sql change): users can pass locality aware backup URIs to SHOW
BACKUP. This change only affects SHOW BACKUP with the new syntax: e.g.:

SHOW BACKUP FROM LATEST IN (<collectionURI>, <localityURI1>, <localityURI2>)

Users can not yet run SHOW BACKUP for locality aware backups created using the
incremental_location parameter. I'm hoping to slip this into the 22.1 release!
msbutler added a commit to msbutler/cockroach that referenced this issue Apr 18, 2022
@msbutler
backupccl: allow user to run SHOW BACKUP on locality aware backups
13d339a 
Previously, the user could only run  SHOW BACKUP on the default locality of a
backup. This patch allows users to pass in all backup localities, allowing
future changes to SHOW BACKUP (e.g. cockroachdb#77694) to analyze data from backup files
in non default locaties.

Users can not yet run SHOW BACKUP for locality aware backups created using the
incremental_location parameter.

Informs cockroachdb#78632

Release note (sql change): users can pass locality aware backup URIs to SHOW
BACKUP. This change only affects SHOW BACKUP with the new syntax: e.g.:

SHOW BACKUP FROM LATEST IN (<collectionURI>, <localityURI1>, <localityURI2>)

Users can not yet run SHOW BACKUP for locality aware backups created using the
incremental_location parameter.
msbutler added a commit to msbutler/cockroach that referenced this issue Apr 20, 2022
@msbutler
backupccl: allow user to run SHOW BACKUP on locality aware backups
acf60fb 
Previously, the user could only run  SHOW BACKUP on the default locality of a
backup. This patch allows users to pass in all backup localities, allowing
future changes to SHOW BACKUP (e.g. cockroachdb#77694) to analyze data from backup files
in non default locaties.

Users can not yet run SHOW BACKUP for locality aware backups created using the
incremental_location parameter.

Informs cockroachdb#78632

Release note (sql change): users can pass locality aware backup URIs to SHOW
BACKUP. This change only affects SHOW BACKUP with the new syntax: e.g.:

SHOW BACKUP FROM LATEST IN (<collectionURI>, <localityURI1>, <localityURI2>)

Users can not yet run SHOW BACKUP for locality aware backups created using the
incremental_location parameter.
craig bot pushed a commit that referenced this issue Apr 20, 2022
@msbutler
Merge #79121
8a45881 
79121: backupccl: allow user to run SHOW BACKUP on locality aware backups r=adityamaru,benbardin a=msbutler

Previously, the user could only run  SHOW BACKUP on the default locality of a
backup. This patch allows users to pass in all backup localities, allowing
future changes to SHOW BACKUP (e.g. #77694) to analyze data from backup files
in non default locaties.

Users can not yet run SHOW BACKUP for locality aware backups created using the
incremental_location parameter.

Informs #78632

Release note (sql change): users can pass locality aware backup URIs to SHOW
BACKUP. This change only affects SHOW BACKUP with the new syntax: e.g.:

SHOW BACKUP FROM LATEST IN (<collectionURI>, <localityURI1>, <localityURI2>)

Users can not yet run SHOW BACKUP for locality aware backups created using the
incremental_location parameter.

Co-authored-by: Michael Butler <[email protected]>
@msbutler msbutler mentioned this issue Apr 26, 2022
Merged
@craig craig bot closed this as completed in 2b8443d May 5, 2022
@mwang1026
Copy link

sync done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@msbutler msbutler

Labels
A-disaster-recovery branch-release-22.1 Used to mark GA and release blockers, technical advisories, and bugs for 22.1 C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) T-disaster-recovery
Projects
No open projects
Disaster Recovery Backlog
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

backupccl: add 'check_files' option to SHOW BACKUP msbutler/cockroach
2 participants
@mwang1026 @msbutler