backupccl: display up to 10 missing files in SHOW BACKUP .. with check_files
#82274
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This change is |
msbutler
force-pushed
the
butler-show-check-return-files
branch
3 times, most recently
from
3438a3c to
5b5d819
Compare
dt
reviewed
Jun 7, 2022
dt
approved these changes
Jun 7, 2022
…ck_files` Previously, `SHOW BACKUP WITH check_files` displayed the first missing SST. This patch will display up to 100 missing SSTs. Further, this renames the misleading `approximateTablePhysicalSize` to `approximateSpanPhysicalSize`. Below I write out how physical table size is calculated: 1. Each range we backup maps to 1 to many spans (currently in the backup_manfest.files object). 2. 1 to many spans get written to an SST. No span will get written to multiple SSTs. 3. When backup created these spans, it tried really hard to split spans at table boundaries, so only one table’s data could be in a span, but a last minute table creation makes this near impossible, due to slow range splits. A big table will have many spans. 4. To compute the approximate logical size (called size_bytes in SHOW BACKUP) of each table, we sum the logical bytes over all it’s spans. We identify a table’s span by checking the table prefix of the first key in the span. See getTableSizes method) 5. To compute the physical size (file_bytes in SHOW BACKUP) of a span, compute the logical size of each SST by summing the logical bytes in the SST over its spans (see getLogicalSSTSize method), and attribute a portion of the physical SST size (returned from cloud storage) to a span using the formula: (sstPhysicalSize) * (logicalSpanSize) / (logicalSSTSize) = physicalSpanSize ( the approximateSpanTableSize method implements this). 6. To compute the physical size of a table, sum over the physical sizes the table’s spans Release note (sql change): SHOW BACKUP WITH check_files will display up to 10 missing SSTs.
msbutler
force-pushed
the
butler-show-check-return-files
branch
from
b17760d to
509e4a0
Compare
|
bors r=dt |
|
Build succeeded: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously,
SHOW BACKUP WITH check_filesdisplayed the first missing SST.This patch will display up to 100 missing SSTs. Further, this renames the
misleading
approximateTablePhysicalSizetoapproximateSpanPhysicalSize.Below I write out how physical table size is calculated:
Each range we backup maps to 1 to many spans (currently in the
backup_manfest.files object).
1 to many spans get written to an SST. No span will get written to multiple
SSTs.
When backup created these spans, it tried really hard to split spans at
table boundaries, so only one table’s data could be in a span, but a last
minute table creation makes this near impossible, due to slow range splits.
A big table will have many spans.
To compute the approximate logical size (called size_bytes in SHOW BACKUP)
of each table, we sum the logical bytes over all it’s spans. We identify a
table’s span by checking the table prefix of the first key in the span. See
getTableSizes method)
To compute the physical size (file_bytes in SHOW BACKUP) of a span, compute
the logical size of each SST by summing the logical bytes in the SST over its
spans (see getLogicalSSTSize method), and attribute a portion of the physical
SST size (returned from cloud storage) to a span using the formula:
(sstPhysicalSize) * (logicalSpanSize) / (logicalSSTSize) = physicalSpanSize (
the approximateSpanTableSize method implements this).
To compute the physical size of a table, sum over the physical sizes the
table’s spans
This patch cleans up code from #80491
Release note (sql change): SHOW BACKUP WITH check_files will display up to 10
missing SSTs.