db: have Checkpoint support shared/external files #3362
Assignees
Labels
Comments
itsbilal
added a commit
to itsbilal/pebble
that referenced
this issue
Mar 15, 2024
Previously, if we couldn't read a remote file's DiskFileNum locally, we'd error out on the Checkpoint() function even though the store was valid and the file just happened to be remote (shared or external). This change adds support for saving the catalog's state to the checkpoint directory so references to remote files from the checkpoint can still be resolved. To prevent unreffing of shared files, we need to add a dangling reference to them that doesn't get deleted. Fixes cockroachdb#3362.
itsbilal
added a commit
to itsbilal/pebble
that referenced
this issue
Mar 18, 2024
Previously, if we couldn't read a remote file's DiskFileNum locally, we'd error out on the Checkpoint() function even though the store was valid and the file just happened to be remote (shared or external). This change adds support for saving the catalog's state to the checkpoint directory so references to remote files from the checkpoint can still be resolved. To prevent unreffing of shared files, we need to add a dangling reference to them that doesn't get deleted. Fixes cockroachdb#3362.
itsbilal
added a commit
to itsbilal/pebble
that referenced
this issue
Mar 18, 2024
Previously, if we couldn't read a remote file's DiskFileNum locally, we'd error out on the Checkpoint() function even though the store was valid and the file just happened to be remote (shared or external). This change adds support for saving the catalog's state to the checkpoint directory so references to remote files from the checkpoint can still be resolved. To prevent unreffing of shared files, we need to add a dangling reference to them that doesn't get deleted. Fixes cockroachdb#3362.
itsbilal
added a commit
that referenced
this issue
Mar 18, 2024
Previously, if we couldn't read a remote file's DiskFileNum locally, we'd error out on the Checkpoint() function even though the store was valid and the file just happened to be remote (shared or external). This change adds support for saving the catalog's state to the checkpoint directory so references to remote files from the checkpoint can still be resolved. To prevent unreffing of shared files, we need to add a dangling reference to them that doesn't get deleted. Fixes #3362.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Currently, we don't copy over or link shared/external files when
Checkpointis called. This is broadly desirable as Checkpoint is supposed to be a quick operation that is often called as part of a crash-worthy event (eg. replica corruption), and copying over gigabites of sstables from remote storage before crashing sounds undesirable.As part of this change, we should ensure Pebble also copies over the objprovider Catalog so any references to external/shared files are copied over. Furthermore, we should ensure that we leave any references on shared files dangling after a Checkpoint so those files don't get deleted by other nodes.
The text was updated successfully, but these errors were encountered: