Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ensure quick-repair commit on shutdown #897

Merged
merged 1 commit into from
Nov 17, 2024

Conversation

mconst
Copy link
Contributor

@mconst mconst commented Nov 17, 2024

Traditionally, redb has stored its allocator state in the region headers and region tracker page, which are only valid after a clean shutdown. Now that we also have the allocator state table (which works even after a crash), it would be nice to use that exclusively.

This is the first step in that transition: making sure that after a clean shutdown, the allocator state is saved to both places. Then, once enough time has passed, we can remove all support for reading and writing the old allocator state.

Even after the transition, any redb version will be able to open any database. If the database has been opened at least once during the transition period, it'll open cleanly; otherwise it'll require repair, but it'll always work.

Note that this means we're saving the allocator state twice on shutdown (unless the user's last commit already had quick-repair enabled), which costs maybe 50 ms for a small database or a few hundred ms for a large one. (But large databases probably want to use quick-repair anyway, in which case there's no cost at all.) It's most noticeable when running the tests, which already spend a substantial fraction of their time writing out allocator state, since they're constantly opening and closing databases without writing much actual data.

Most of this cost will go away in the future when we get rid of the old allocator state, but not all. At the moment, writing the allocator state table is a bit slower than writing the old allocator state, mostly because of inefficiencies in how redb handles large pages. I haven't been worrying about this, since I feel like the performance is totally acceptable already -- but if you're interested, let me know and I can tell you all about what could be done to make it faster. None of this involves breaking compatibility, so these improvements could be done at any point.

Traditionally, redb has stored its allocator state in the region headers
and region tracker page, which are only valid after a clean shutdown.
Now that we also have the allocator state table (which works even after
a crash), it would be nice to use that exclusively.

This is the first step in that transition: making sure that after a clean
shutdown, the allocator state is saved to both places. Then, once enough
time has passed, we can remove all support for reading and writing the
old allocator state.

Even after the transition, any redb version will be able to open any
database. If the database has been opened at least once during the
transition period, it'll open cleanly; otherwise it'll require repair,
but it'll always work.
@cberner cberner merged commit b879f35 into cberner:master Nov 17, 2024
3 checks passed
@mconst mconst deleted the ensure_quick_repair branch November 18, 2024 08:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants