[Merged by Bors] - Implement checkpoint sync #2244

michaelsproul · 2021-03-05T06:35:50Z

Issue Addressed

Closes #1891
Closes #1784

Proposed Changes

Implement checkpoint sync for Lighthouse, enabling it to start from a weak subjectivity checkpoint.

Additional Info

Return unavailable status for out-of-range blocks requested by peers (Network Update for Weak Subjectivity Sync #2561)
Implement sync daemon for fetching historical blocks (Network Update for Weak Subjectivity Sync #2561)
Verify chain hashes (either in historical_blocks.rs or the calling module)
Consistency check for initial block + state
Fetch the initial state and block from a beacon node HTTP endpoint
Don't crash fetching beacon states by slot from the API
Background service for state reconstruction, triggered by CLI flag or API call.

Considered out of scope for this PR:

Drop the requirement to provide the --checkpoint-block (this would require some pretty heavy refactoring of block verification)

beacon_node/beacon_chain/src/historical_blocks.rs

paulhauner · 2021-08-02T01:59:16Z

I've tagged this for v1.5.1. That might be optimistic, but I just wanted to keep this on the radar. Feel free to downgrade :)

michaelsproul · 2021-08-05T07:58:03Z

I've made some tweaks to help with debugging:

Update lcli pretty-ssz to print JSON, and support SignedBeaconBlock
Add two new /lighthouse/database endpoints:
- /info: info about the split slot, anchor point, and schema version
- /historical_blocks: POST endpoint for pushing historical blocks to a BN manually

My debugging flow (using scripts from eth2-scripts) has been:

$ ssz_block.sh finalized > block.ssz
$ ssz_state.sh finalized > state.ssz
$ lighthouse bn --initial-state state.ssz --initial-block block.ssz --http
$ historical_sync.py

Now to fix the actual problems listed in the PR description 😅

michaelsproul · 2021-08-06T08:33:10Z

There's an issue with the Altair beacon chain tests. Probably something benign, but I haven't had a chance to look into it yet.

michaelsproul · 2021-08-14T05:05:24Z

Also realised there are probably issues at skipped slots. If the initial block's slot doesn't match the state's I think we will run into problems. I'll look into the best way to handle this next week.

* Close to first draft * Further progress to first draft * Further progress, before rebase * First draft * Cleanup and fixes * Notifier updates and bug fixes * Fix off-by-one errors * Remove todo * Increase backfill buffer * Gracefully handle requests during backfill * Improve comments * fmt * Update error handling * Reviewers suggestions * Take historic blocks by ref, avoid clone * Clear batch on pause * Further reviewers suggestions Co-authored-by: Michael Sproul <[email protected]>

michaelsproul · 2021-09-14T23:34:21Z

This is ready for review! Batteries now included 🎉

* Give error when WS sync with existing DB * Give wrong network hint for bad SSZ * Use checkpoint instead of check-point

* Log about backfill less often * Log when backfill complete * Update docs

paulhauner

This is such an awesome feature, I had so much fun using it 🎉

I've left a few rather minor suggestions and comments. I also have a few that don't relate to changes so I can't associate to a specific line:

The --shutdown-after-sync flag will now shutdown after we reach head, but not before we finish back fill. I don't really know what the behaviour should be here, but at the very least we could add a message to the CLI help string flag indicating that this is the case.
We also have the --wss-checkpoint flag on the BN. I think it should probably conflict with all the checkpoint-sync flags.

I also noticed we don't have the client-side of the new /lighthouse API endpoints. It would be fairly painless to add these and some smoke tests. I might pick this up when I get a chance and make a PR to this PR.

beacon_node/beacon_chain/src/schema_change.rs

beacon_node/network/src/sync/backfill_sync/mod.rs

beacon_node/store/src/reconstruct.rs

book/src/checkpoint-sync.md

consensus/state_processing/src/per_block_processing/block_signature_verifier.rs

beacon_node/network/src/sync/manager.rs

beacon_node/network/src/sync/backfill_sync/mod.rs

michaelsproul · 2021-09-20T05:43:47Z

Thanks @paulhauner for the review, and thanks @AgeManning for fixing up that sync TODO. All comments have now been addressed, so I think we must be almost ready to merge 🎉

michaelsproul · 2021-09-20T05:50:50Z

Ah, I didn't address the flag-related suggestions from your main comment. Will do that now.

michaelsproul · 2021-09-20T06:09:06Z

Done.

Points to note:

Deleted the /lighthouse/backfill API because its functionality was covered by lighthouse/syncing. Age was just using it during testing.
The lighthouse/database/historical_blocks API is untested because it would be a pain to do a non-trivial test (requires a WSS node, and some blocks to POST). Seeing as it's for developers only and the docs state as much, I think this is an acceptable tradeoff.
I've left --wss-checkpoint and --checkpoint-* as non-conflicting, because I think there are cases in which it could be useful to have both, e.g. if you start a checkpoint sync from several epochs back but want to verify that you reach some more recent checkpoint. This can be useful when you want quick access to some recent history, without having to backfill + reconstruct. I've updated the help text to point to --checkpoint-sync-url.

paulhauner

Awesome, I can't wait to see this in the hands of users!

You have my approval, merge at will capt'n 🎉

michaelsproul · 2021-09-22T00:37:12Z

😮 😮 😮

bors r+

## Issue Addressed Closes #1891 Closes #1784 ## Proposed Changes Implement checkpoint sync for Lighthouse, enabling it to start from a weak subjectivity checkpoint. ## Additional Info - [x] Return unavailable status for out-of-range blocks requested by peers (#2561) - [x] Implement sync daemon for fetching historical blocks (#2561) - [x] Verify chain hashes (either in `historical_blocks.rs` or the calling module) - [x] Consistency check for initial block + state - [x] Fetch the initial state and block from a beacon node HTTP endpoint - [x] Don't crash fetching beacon states by slot from the API - [x] Background service for state reconstruction, triggered by CLI flag or API call. Considered out of scope for this PR: - Drop the requirement to provide the `--checkpoint-block` (this would require some pretty heavy refactoring of block verification) Co-authored-by: Diva M <[email protected]>

bors · 2021-09-22T02:29:56Z

Pull request successfully merged into unstable.

Build succeeded:

michaelsproul added 6 commits February 23, 2021 11:25

Implement starting from a weak subectivity state

719a934

Support for historical blocks

4dcf2e7

Fix iterators and StatusMessage processing

3aa7c7b

Merge remote-tracking branch 'origin/unstable' into weak-subj-sync

366ae60

Implement schema migration

d8e8d97

Remove some TODOs/FIXMEs

6cdac53

michaelsproul added the work-in-progress PR is a work-in-progress label Mar 5, 2021

michaelsproul self-assigned this Mar 5, 2021

Clean up APIs, fix several TODOs

22867d7

michaelsproul mentioned this pull request Apr 30, 2021

Implement 2 minutes beacon node sync with checkpoint state #2328

Closed

Merge branch 'unstable' into weak-subj-sync

dd55207

michaelsproul commented Jul 16, 2021

View reviewed changes

beacon_node/beacon_chain/src/historical_blocks.rs Outdated Show resolved Hide resolved

paulhauner added the v1.5.1 To be included in the v1.5.1 relase label Aug 2, 2021

michaelsproul added 5 commits August 4, 2021 15:31

Merge remote-tracking branch 'origin/unstable' into weak-subj-sync

f86032f

Fixes for merge issues, block separation

0066231

Add missed comma

25306fc

Improve lcli pretty-ssz & logging

6cdb165

Add database HTTP APIs

20a97a4

michaelsproul added 2 commits August 6, 2021 11:43

Fix clippy

8d812ce

Fix block_root_at_slot for historic blocks

d966be6

michaelsproul added 5 commits August 16, 2021 14:17

Add consistency checks

5221ac6

Fix Altair test by setting spec

9b290f8

Add --checkpoint-sync-url

6b79baf

Merge remote-tracking branch 'origin/unstable' into weak-subj-sync

5e92a7a

Update Cargo.lock

8865e90

michaelsproul changed the title ~~Implement weak subjectivity sync~~ Implement checkpoint sync Aug 16, 2021

michaelsproul added ready-for-review The code is ready for review and removed blocked labels Sep 14, 2021

michaelsproul requested a review from paulhauner September 14, 2021 23:33

paulhauner added 2 commits September 15, 2021 21:29

Additional checkpoint sync logs (#2599)

81ed51a

* Give error when WS sync with existing DB * Give wrong network hint for bad SSZ * Use checkpoint instead of check-point

Log about historical backfill less frequently (#2600)

8e32a1e

* Log about backfill less often * Log when backfill complete * Update docs

michaelsproul mentioned this pull request Sep 17, 2021

[Merged by Bors] - Improved handling of IP Banning #2530

Closed

paulhauner requested changes Sep 17, 2021

View reviewed changes

paulhauner added waiting-on-author The reviewer has suggested changes and awaits thier implementation. and removed ready-for-review The code is ready for review labels Sep 17, 2021

AgeManning and others added 2 commits September 20, 2021 14:57

Don't downscore peers in batch sync when disconnecting

b404f4d

Address review comments

30e2219

michaelsproul added ready-for-review The code is ready for review and removed waiting-on-author The reviewer has suggested changes and awaits thier implementation. labels Sep 20, 2021

Tweak CLI flag help text

6dbedfe

AgeManning and others added 2 commits September 20, 2021 20:14

Improved chain resuming

fc28aeb

Fix clippy

c85c8ab

paulhauner approved these changes Sep 22, 2021

View reviewed changes

paulhauner added ready-for-merge This PR is ready to merge. and removed ready-for-review The code is ready for review labels Sep 22, 2021

bors bot changed the title ~~Implement checkpoint sync~~ [Merged by Bors] - Implement checkpoint sync Sep 22, 2021

bors bot closed this Sep 22, 2021

michaelsproul deleted the weak-subj-sync branch September 22, 2021 03:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Merged by Bors] - Implement checkpoint sync #2244

[Merged by Bors] - Implement checkpoint sync #2244

michaelsproul commented Mar 5, 2021 •

edited

Loading

paulhauner commented Aug 2, 2021

michaelsproul commented Aug 5, 2021

michaelsproul commented Aug 6, 2021

michaelsproul commented Aug 14, 2021

michaelsproul commented Sep 14, 2021

paulhauner left a comment

michaelsproul commented Sep 20, 2021

michaelsproul commented Sep 20, 2021 •

edited

Loading

michaelsproul commented Sep 20, 2021

paulhauner left a comment

michaelsproul commented Sep 22, 2021 •

edited

Loading

bors bot commented Sep 22, 2021

[Merged by Bors] - Implement checkpoint sync #2244

[Merged by Bors] - Implement checkpoint sync #2244

Conversation

michaelsproul commented Mar 5, 2021 • edited Loading

Issue Addressed

Proposed Changes

Additional Info

paulhauner commented Aug 2, 2021

michaelsproul commented Aug 5, 2021

michaelsproul commented Aug 6, 2021

michaelsproul commented Aug 14, 2021

michaelsproul commented Sep 14, 2021

paulhauner left a comment

Choose a reason for hiding this comment

michaelsproul commented Sep 20, 2021

michaelsproul commented Sep 20, 2021 • edited Loading

michaelsproul commented Sep 20, 2021

paulhauner left a comment

Choose a reason for hiding this comment

michaelsproul commented Sep 22, 2021 • edited Loading

bors bot commented Sep 22, 2021

michaelsproul commented Mar 5, 2021 •

edited

Loading

michaelsproul commented Sep 20, 2021 •

edited

Loading

michaelsproul commented Sep 22, 2021 •

edited

Loading