Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Handle ungraceful shutdown on first startup #2067

Open
michaelsproul opened this issue Dec 8, 2020 · 0 comments
Open

Handle ungraceful shutdown on first startup #2067

michaelsproul opened this issue Dec 8, 2020 · 0 comments
Labels
A1 database

Comments

@michaelsproul
Copy link
Member

Description

I think there's an issue where if the beacon node is killed or shutdown ungracefully before it has first written the head* and fork choice to disk, then it will be unable to resume.

I suspect dealing with this 100% reliably would be quite difficult, unless we are able to wrap the creation of the database, and all of its initial entries in one big atomic transaction. Incremental improvements may be better, but would also require more thought.

This issue is similar to #1680 and #1686

*NB: The head is actually just the head tracker and genesis block these days (see #1784)
@michaelsproul michaelsproul mentioned this issue Dec 8, 2020
Closed
bors bot pushed a commit that referenced this issue Dec 9, 2020
@michaelsproul
Revert fork choice if disk write fails (#2068)
aa45fa3 
## Issue Addressed

Closes #2028
Replaces #2059

## Proposed Changes

If writing to the database fails while importing a block, revert fork choice to the last version stored on disk. This prevents fork choice from being ahead of the blocks on disk. Having fork choice ahead is particularly bad if it is later successfully written to disk, because it renders the database corrupt (see #2028).

## Additional Info

* This mitigation might fail if the head+fork choice haven't been persisted yet, which can only happen at first startup (see #2067)
* This relies on it being OK for the head tracker to be ahead of fork choice. I figure this is tolerable because blocks only get added to the head tracker after successfully being written on disk _and_ to fork choice, so even if fork choice reverts a little bit, when the pruning algorithm runs, those blocks will still be on disk and OK to prune. The pruning algorithm also doesn't rely on heads being unique, technically it's OK for multiple blocks from the same linear chain segment to be present in the head tracker. This begs the question of #1785 (i.e. things would be simpler with the head tracker out of the way). Alternatively, this PR could just revert the head tracker as well (I'll look into this tomorrow).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A1 database
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@michaelsproul