[persist] Design for backup/restore #18157
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
5 tasks
bkirwi
added a commit
that referenced
this pull request
Oct 19, 2023
Persist state is a mix of stuff in CRDB and stuff in S3. This complicates restoring from backup: if we restore CRDB to an old state, the blobs it references might be deleted. It's possible to "undelete" deleted blobs in S3 by deleting a delete marker from an unversioned bucket. This PR implements that logic as a CLI tool, and implements a basic test that restores a CRDB state and checks that the database is in the expected state after the CLI runs. ### Motivation See #18157 for the design. ### Tips for reviewer One subtlety here is that we can't rely on the usual state iterators, since those rely on the existence of rollups, which may not exist yet. The current draft just iterates over the diffs manually. I have not yet tested this tool against a real environment. I will probably wait for at least a first pass of code review first! Co-authored-by: Philip Stoev <[email protected]>
danhhz
reviewed
Nov 2, 2023
|
Oops, thanks for catching that! No pressing need; just doing some housekeeping. (This predates the new design process, so the schedule's a bit loose, but anyways it feels like it's ready to go.) |
danhhz
approved these changes
Nov 3, 2023
matthewarthur
pushed a commit
that referenced
this pull request
Nov 7, 2023
Known-desirable: see #17605 for context. ### Tips for reviewer [Click here for the rendered version.](https://github.com/bkirwi/materialize/blob/backup-design/doc/developer/design/20230315_backup_restore.md) I've brought this closer to the standard design doc template, and filled in a few details that were left open earlier this year.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Known-desirable: see MaterializeInc/database-issues#5119 for context.
Tips for reviewer
Click here for the rendered version.
Edit: I've brought this closer to the standard design doc template, and filled in a few details that were left open earlier this year.
Checklist
$T ⇔ Proto$Tmapping (possibly in a backwards-incompatible way) and therefore is tagged with aT-protolabel.