Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Merged by Bors] - Realized unrealized experimentation #3322

Closed
wants to merge 179 commits into from
Closed

[Merged by Bors] - Realized unrealized experimentation #3322

wants to merge 179 commits into from

Conversation

realbigsean
Copy link
Member

Issue Addressed

Add a flag that optionally enables unrealized vote tracking. Would like to test out on testnets and benchmark differences in methods of vote tracking. This PR includes a DB schema upgrade to enable to new vote tracking style.

realbigsean and others added 30 commits April 23, 2022 09:04
@realbigsean
separate participation cache by epoch and add it to the beacon state
5cae2f9
@realbigsean
Mutate state via mini beacon state in epoch processing
6c7ab85
@realbigsean
fork choice schema migration
fb16cd7
@realbigsean
update filter block tree
24a5c16
@realbigsean
cargo fmt and fix
24c6273
@realbigsean
refactors
513d2b2
@realbigsean
fixes and debugging
1b81c56
@realbigsean
debugging
e67f07d
@realbigsean
remove debugging lines, fix chache initialization slot
1854a23
@paulhauner @realbigsean
Add first unit test
365d58c
@realbigsean
adjust pauls test to only run for altair and to, and to assert the he…
2ae43cd 
…ad remains unchanged
@realbigsean
Add scenario for n + 2
8a5d9f3
@realbigsean
Merge branch 'unstable' of https://github.com/sigp/lighthouse into fc…
e166d55 
…-refactor

# Conflicts:
#	beacon_node/beacon_chain/src/schema_change.rs
#	beacon_node/beacon_chain/src/schema_change/migration_schema_v9.rs
#	beacon_node/http_api/src/attestation_performance.rs
#	consensus/types/src/beacon_state/participation_cache.rs
@realbigsean
updates for fc store unrealized justification tracking
dc51a24
@realbigsean
filter block tree changes
da9047c
@realbigsean
changes for tests
58b2892
@realbigsean
Merge branch 'unstable' of https://github.com/sigp/lighthouse into fc…
0ce2a6b 
…-refactor
@realbigsean
test
3a0fea7
@realbigsean
revert changes to ef test tester
76b5344
@realbigsean
remove ef-test proposer boost shim
5b43461
@realbigsean
- pass thrugh spec argument
4ce3e40 
- parse discard_equivocations test
@realbigsean
EthSpec generic in proto array
c8315e1
@realbigsean
process justifiable for base as well
2421281
@realbigsean
Merge branch 'fc-refactor' of https://github.com/realbigsean/lighthouse…
a7b1e82 
… into fc-refactor
@realbigsean
update justification and finalization method
7d8f85f
@macladson @paulhauner
Squash previous works
ec5f058 
Update HTTP API to include `execution_optimistic` flag in responses

Add `execution_optimistic` to server-sent events

Fix formatting

Fix tests

Fix endpoint comment

Remove redundent calls to `get_block`

Fix route

Return correct `execution_optimistic` values for api queries by state

Updated api client functions

Refactor HTTP API tests

Return 404 when requested root is 0x000..000

Tidy up

Avoid recomputing head

More improvements

Use head when computing status for slot

Squash several commit for 3175

Start adding ChainView

Ripple `ChainView` across code-base

Fix after rebase on sigp#3070

Pass `make lint` with `chain_view`

Remove uses of get_head

Remove uses of `with_head`

Remove use of canonical head

Start combining head and fork choice

Big refactor to further lock joining

Fix to pass `make lint`

Fix consensus/fork_choice tests

Fix http_api tests

Remove current_slot from FC rebuild

Fix lint issues after rebase

Refactor out into recompute_head.rs

Add TaskExecutor to BeaconChainHarness

Thread task_executor into BeaconChain

Use async fork choice

Fix test compile errors

Clear early attester cache sooner

Move canonical_head to own file, fix test deadlock

Get `make lint` working again

Add `genesis_time` to `BeaconChain`

Tidy

Fix test compile issues

Move recompute_head fns together

Spawn EL update tasks

Move EL updates into own function

Tidy
@paulhauner
Resolve BeaconChain compile errors
61ff3b5
@paulhauner
Revert common/eth2 to unstable
4fa5be6
@paulhauner
Fix http_api to pass make lint
ac01a7e
@paulhauner
Remove old fork choice functions
8ade87e
@michaelsproul michaelsproul added the backwards-incompat Backwards-incompatible API change label Jul 15, 2022
@realbigsean
Merge branch 'unstable' of https://github.com/sigp/lighthouse into re…
58e8134 
…alized-unrealized-experimentation

� Conflicts:
�	beacon_node/network/src/beacon_processor/worker/gossip_methods.rs
@realbigsean
lints
be4955e
@paulhauner paulhauner added the v2.5.0 Required for Goerli merge release label Jul 23, 2022
paulhauner
paulhauner approved these changes Jul 23, 2022
View reviewed changes
Copy link
Member

@paulhauner paulhauner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome work, good to go! It's great to see we have a new fork choice expert on our hands 🎉

I had some awkward times with Github; I was reviewing an old commit or something and ended up making some invalid comments. Hopefully they don't show up, I'll prune them if they do.

Let's give v2.4.0 a little time settle before we merge this. Let's catch up on your Monday afternoon (my Tuesday morning) and then merge it then 🚀

beacon_node/beacon_chain/src/schema_change/migration_schema_v10.rs Show resolved Hide resolved
@paulhauner paulhauner added ready-for-merge This PR is ready to merge. and removed ready-for-review The code is ready for review labels Jul 23, 2022
@paulhauner
Copy link
Member

I'll leave this one to @realbigsean to merge when he's ready ☺️

michaelsproul added a commit to michaelsproul/lighthouse that referenced this pull request Jul 25, 2022
@michaelsproul
[temp] Realized unrealized experimentation (sigp#3322)
f4a2048 
Squashed commit of the following:

commit be4955e
Author: realbigsean <[email protected]>
Date:   Thu Jul 21 20:05:05 2022 -0400

    lints

commit 58e8134
Merge: 7265479 612cdb7
Author: realbigsean <[email protected]>
Date:   Thu Jul 21 19:59:46 2022 -0400

    Merge branch 'unstable' of https://github.com/sigp/lighthouse into realized-unrealized-experimentation

    � Conflicts:
    �	beacon_node/network/src/beacon_processor/worker/gossip_methods.rs

commit 7265479
Author: realbigsean <[email protected]>
Date:   Wed Jul 13 21:09:14 2022 -0400

    hide -> hidden

commit 04647be
Author: realbigsean <[email protected]>
Date:   Wed Jul 13 18:37:02 2022 -0400

    fix compile error

commit 89c27d0
Author: realbigsean <[email protected]>
Date:   Wed Jul 13 14:48:37 2022 -0400

    iter

commit 3e3b9e0
Author: Paul Hauner <[email protected]>
Date:   Wed Jul 13 18:49:27 2022 +1000

    Add late-epoch optimisation

commit ec06f03
Author: Paul Hauner <[email protected]>
Date:   Wed Jul 13 18:18:07 2022 +1000

    Add `UpdateJustifiedCheckpointSlots`

commit 8496857
Author: realbigsean <[email protected]>
Date:   Wed Jul 13 10:48:31 2022 -0400

    cargo fmt

commit 0c6de63
Author: realbigsean <[email protected]>
Date:   Wed Jul 13 10:43:57 2022 -0400

    cargo fmt

commit b3ef232
Author: realbigsean <[email protected]>
Date:   Wed Jul 13 10:41:33 2022 -0400

    only count unrealized in last block of chain in block replay

commit e01fa88
Author: realbigsean <[email protected]>
Date:   Wed Jul 13 10:04:53 2022 -0400

    Update testing/ef_tests/src/cases/fork_choice.rs

    Co-authored-by: Paul Hauner <[email protected]>

commit 3c80207
Author: realbigsean <[email protected]>
Date:   Wed Jul 13 10:04:43 2022 -0400

    Update testing/ef_tests/src/cases/fork_choice.rs

    Co-authored-by: Paul Hauner <[email protected]>

commit 3f9e6c3
Author: realbigsean <[email protected]>
Date:   Wed Jul 13 10:04:27 2022 -0400

    Update consensus/state_processing/src/per_epoch_processing/altair/participation_cache.rs

    Co-authored-by: Paul Hauner <[email protected]>

commit 8fbb9e7
Author: realbigsean <[email protected]>
Date:   Wed Jul 13 10:03:49 2022 -0400

    Update consensus/state_processing/src/per_epoch_processing/justification_and_finalization_state.rs

    Co-authored-by: Paul Hauner <[email protected]>

commit 5e512f7
Author: realbigsean <[email protected]>
Date:   Wed Jul 13 10:03:09 2022 -0400

    Update consensus/fork_choice/src/fork_choice.rs

    Co-authored-by: Paul Hauner <[email protected]>

commit d4269a1
Author: realbigsean <[email protected]>
Date:   Wed Jul 13 10:03:01 2022 -0400

    Update beacon_node/src/cli.rs

    Co-authored-by: Paul Hauner <[email protected]>

commit 828d5bc
Merge: e939bd4 4212f22
Author: realbigsean <[email protected]>
Date:   Tue Jul 12 11:17:03 2022 -0400

    Merge branch 'unstable' of https://github.com/sigp/lighthouse into realized-unrealized-experimentation

commit e939bd4
Author: realbigsean <[email protected]>
Date:   Tue Jul 12 11:16:38 2022 -0400

    copied on Result in unstable in our current msrv

commit 9737b2b
Author: realbigsean <[email protected]>
Date:   Sun Jul 10 17:52:37 2022 -0400

    delete commented out test

commit dc6e836
Author: realbigsean <[email protected]>
Date:   Sun Jul 10 17:14:30 2022 -0400

    fix tests

commit c1a44c7
Author: realbigsean <[email protected]>
Date:   Sun Jul 10 16:41:46 2022 -0400

    fix test

commit b8ce0cb
Author: realbigsean <[email protected]>
Date:   Fri Jul 8 16:02:43 2022 -0400

    Only count unrealized when we are syncing non-finalized chain segments

commit 9951c2f
Author: realbigsean <[email protected]>
Date:   Fri Jul 8 11:04:43 2022 -0400

    get tests passing

commit 1df5164
Merge: d0f6f9e 5dbfb37
Author: realbigsean <[email protected]>
Date:   Fri Jul 8 09:38:09 2022 -0400

    Merge branch 'unstable' of https://github.com/sigp/lighthouse into realized-unrealized-experimentation

commit d0f6f9e
Author: realbigsean <[email protected]>
Date:   Thu Jul 7 10:54:44 2022 -0400

    add flag to enable unrealized vote counting

commit 6b71365
Author: realbigsean <[email protected]>
Date:   Tue Jul 5 19:05:34 2022 -0400

    fix compile errors after merging

commit d935728
Merge: f073510 748475b
Author: realbigsean <[email protected]>
Date:   Tue Jul 5 18:45:52 2022 -0400

    Merge branch 'unstable' of https://github.com/sigp/lighthouse into realized-unrealized-experimentation

    � Conflicts:
    �	beacon_node/beacon_chain/src/beacon_chain.rs
    �	beacon_node/beacon_chain/src/block_verification.rs
    �	beacon_node/beacon_chain/src/canonical_head.rs
    �	beacon_node/beacon_chain/src/execution_payload.rs
    �	beacon_node/beacon_chain/src/schema_change.rs
    �	beacon_node/beacon_chain/src/schema_change/migration_schema_v7.rs
    �	beacon_node/beacon_chain/src/test_utils.rs
    �	beacon_node/http_api/src/lib.rs
    �	beacon_node/http_api/tests/tests.rs
    �	beacon_node/network/src/beacon_processor/worker/sync_methods.rs
    �	common/task_executor/src/lib.rs
    �	common/task_executor/src/metrics.rs
    �	consensus/fork_choice/src/fork_choice.rs
    �	consensus/fork_choice/tests/tests.rs
    �	consensus/types/src/beacon_block.rs
    �	consensus/types/src/beacon_block_body.rs
    �	consensus/types/src/beacon_state/committee_cache/tests.rs
    �	testing/ef_tests/src/cases/fork_choice.rs

commit f073510
Author: realbigsean <[email protected]>
Date:   Tue Jul 5 18:07:09 2022 -0400

    fix failing test

commit aff44e8
Author: realbigsean <[email protected]>
Date:   Tue Jul 5 17:50:29 2022 -0400

    Add schema 10 -> 9 downgrade

commit f57cbbb
Author: realbigsean <[email protected]>
Date:   Tue Jul 5 16:22:01 2022 -0400

    remove n plus 2 test

commit aa44285
Author: Paul Hauner <[email protected]>
Date:   Wed Jun 29 16:32:58 2022 +1000

    Ensure best justified is finalized descendent

commit cbd8237
Author: Paul Hauner <[email protected]>
Date:   Wed Jun 29 15:27:16 2022 +1000

    Fix failing gossip verification test

commit 24b4667
Author: Paul Hauner <[email protected]>
Date:   Wed Jun 29 14:09:51 2022 +1000

    Use fork choice for block import finalization

commit de0a2bb
Author: Paul Hauner <[email protected]>
Date:   Wed Jun 29 12:43:14 2022 +1000

    Remove mut ref

commit d73b74c
Merge: 060c3d1 af1623d
Author: realbigsean <[email protected]>
Date:   Tue Jun 28 19:42:12 2022 -0400

    Merge branch 'fc-refactor-participation-cache' of https://github.com/sigp/lighthouse-private into fc-refactor-participation-cache

commit af1623d
Author: Paul Hauner <[email protected]>
Date:   Wed Jun 29 09:11:16 2022 +1000

    Remove participation_cache.rs

commit 060c3d1
Merge: 1b59be9 ab76e2c
Author: realbigsean <[email protected]>
Date:   Tue Jun 28 18:47:21 2022 -0400

    Merge branch 'fc-refactor-participation-cache' of https://github.com/sigp/lighthouse-private into fc-refactor-participation-cache

commit ab76e2c
Author: Paul Hauner <[email protected]>
Date:   Wed Jun 29 08:40:46 2022 +1000

    Account for `state.get_block_root_at_epoch` errors

commit 1b59be9
Author: realbigsean <[email protected]>
Date:   Tue Jun 28 18:21:15 2022 -0400

    add reorg test

commit 94557f2
Author: realbigsean <[email protected]>
Date:   Tue Jun 28 17:12:07 2022 -0400

    uncomment ef tests

commit eb64294
Author: Paul Hauner <[email protected]>
Date:   Tue Jun 28 18:27:28 2022 +1000

    Tidy, fix compile errors

commit 6764608
Author: Paul Hauner <[email protected]>
Date:   Tue Jun 28 18:05:20 2022 +1000

    Add `JustificationAndFinalizationState`

commit 9212770
Author: Paul Hauner <[email protected]>
Date:   Tue Jun 28 17:16:08 2022 +1000

    Revert state_processing and types/beacon_state

commit d98041e
Author: Paul Hauner <[email protected]>
Date:   Tue Jun 28 14:28:10 2022 +1000

    Fix trivial test compile errors

commit 33d7305
Merge: 49321ec 011818f
Author: Paul Hauner <[email protected]>
Date:   Tue Jun 28 13:55:10 2022 +1000

    Merge branch 'nfd-2' into fc-refactor

commit 49321ec
Merge: 7d8f85f 91b877f
Author: Paul Hauner <[email protected]>
Date:   Tue Jun 28 13:39:28 2022 +1000

    Merge pull request #4 from sigp/fc-refactor-nfd-2

    Squash-merge `nfd-2`

commit 91b877f
Author: Paul Hauner <[email protected]>
Date:   Tue Jun 28 13:35:46 2022 +1000

    Squash-merge `nfd-2`

commit 011818f
Author: Paul Hauner <[email protected]>
Date:   Thu Jun 23 10:31:29 2022 +1000

    Address review comments from @realbigsean

commit acc9aac
Author: Paul Hauner <[email protected]>
Date:   Thu Jun 23 10:28:25 2022 +1000

    Use fc write lock for restore

commit 733192a
Author: Paul Hauner <[email protected]>
Date:   Thu Jun 23 09:43:13 2022 +1000

    Apply suggestions from code review

    Co-authored-by: realbigsean <[email protected]>

commit 5f67350
Author: Paul Hauner <[email protected]>
Date:   Wed Jun 22 19:08:37 2022 +1000

    Tidy

commit b46088d
Author: Paul Hauner <[email protected]>
Date:   Wed Jun 22 18:25:36 2022 +1000

    Tidy

commit 74f8e0b
Author: Paul Hauner <[email protected]>
Date:   Wed Jun 22 17:55:15 2022 +1000

    Tidy

commit de9a72e
Author: Paul Hauner <[email protected]>
Date:   Wed Jun 22 16:49:14 2022 +1000

    Reduce chain segment diff

commit f3ac736
Author: Paul Hauner <[email protected]>
Date:   Wed Jun 22 16:38:07 2022 +1000

    Reduce diff for attestation production

commit 2afe98d
Author: Paul Hauner <[email protected]>
Date:   Wed Jun 22 16:30:22 2022 +1000

    Undo changes to `BeaconChain::import_block`

commit 6b3b2b7
Author: Paul Hauner <[email protected]>
Date:   Wed Jun 22 16:17:50 2022 +1000

    Tidy

commit 19870a8
Author: Paul Hauner <[email protected]>
Date:   Wed Jun 22 11:13:51 2022 +1000

    Remove erroneous log for opt transition block

commit 836ac1c
Author: Paul Hauner <[email protected]>
Date:   Tue Jun 21 19:11:45 2022 +1000

    Use rayon in ef_tests by default

commit 219327b
Author: Paul Hauner <[email protected]>
Date:   Tue Jun 21 19:07:59 2022 +1000

    Remove parking lot from ef_tests

commit 42af8db
Author: Paul Hauner <[email protected]>
Date:   Tue Jun 21 19:06:52 2022 +1000

    Remove phantom block_efficiency file

commit 9ca7d49
Author: Paul Hauner <[email protected]>
Date:   Tue Jun 21 18:53:38 2022 +1000

    Update block_on_dangerous comments

commit 934a147
Author: Paul Hauner <[email protected]>
Date:   Tue Jun 21 18:38:32 2022 +1000

    Add comment about val count caches

commit 6de54f1
Author: Paul Hauner <[email protected]>
Date:   Tue Jun 21 18:32:06 2022 +1000

    Reduce duplicate code in sync_methods

commit 343aa37
Author: Paul Hauner <[email protected]>
Date:   Tue Jun 21 18:06:30 2022 +1000

    Add comments to `TaskSpawner`

commit 9d73de2
Author: Paul Hauner <[email protected]>
Date:   Tue Jun 21 17:59:52 2022 +1000

    Remove commented-out code from beacon_processor

commit 57d62f1
Author: Paul Hauner <[email protected]>
Date:   Tue Jun 21 17:28:07 2022 +1000

    Update opt sync error message

commit 2b1bb42
Author: Paul Hauner <[email protected]>
Date:   Tue Jun 21 15:57:09 2022 +1000

    Use blocking task for FC notifier tx

commit b5388ae
Author: Paul Hauner <[email protected]>
Date:   Tue Jun 21 15:53:11 2022 +1000

    Fix conflict in `state_advance_timer`

commit 0d8a9d8
Author: Paul Hauner <[email protected]>
Date:   Tue Jun 21 15:52:59 2022 +1000

    Refactor `BeaconChain::best_slot`

commit fcbe53f
Author: Paul Hauner <[email protected]>
Date:   Tue Jun 21 15:43:57 2022 +1000

    Fix state_advance_timer TODO

commit 2905a9d
Author: Paul Hauner <[email protected]>
Date:   Tue Jun 21 14:12:03 2022 +1000

    Fix broken store test

commit 459c677
Author: Paul Hauner <[email protected]>
Date:   Tue Jun 21 11:43:02 2022 +1000

    Refactor head accessor methods

commit 01e760f
Author: Paul Hauner <[email protected]>
Date:   Tue Jun 21 10:02:07 2022 +1000

    Revert changes to beacon_chain Cargo.toml

commit 7c5ea7a
Author: Paul Hauner <[email protected]>
Date:   Tue Jun 21 10:01:15 2022 +1000

    Revert changes to clippy.toml

commit 44a75e7
Author: Paul Hauner <[email protected]>
Date:   Tue Jun 21 09:47:33 2022 +1000

    Fix flipped bool

commit b5043b9
Author: Paul Hauner <[email protected]>
Date:   Tue Jun 21 09:35:03 2022 +1000

    Unify is_optimistic_candidate_block

commit 858db36
Author: Paul Hauner <[email protected]>
Date:   Mon Jun 20 17:22:34 2022 +1000

    Update Cargo.lock

commit 82d8068
Author: Paul Hauner <[email protected]>
Date:   Mon Jun 20 16:52:31 2022 +1000

    Fix test compile errors

commit 08082fb
Author: Paul Hauner <[email protected]>
Date:   Mon Jun 20 16:21:41 2022 +1000

    Tidy

commit dbde4bf
Author: Paul Hauner <[email protected]>
Date:   Mon Jun 20 15:53:32 2022 +1000

    Expose fork choice locks

commit 7cc0d27
Author: Paul Hauner <[email protected]>
Date:   Mon Jun 20 15:10:01 2022 +1000

    Switch to `Arc<Snapshot>` for `CachedHead`

commit 65ead81
Author: Paul Hauner <[email protected]>
Date:   Fri Jun 17 20:27:44 2022 +1000

    Tidy comments

commit 5d4ad4e
Author: Paul Hauner <[email protected]>
Date:   Fri Jun 17 20:23:06 2022 +1000

    Tidy, add comments

commit 79aab62
Author: Paul Hauner <[email protected]>
Date:   Fri Jun 17 19:36:26 2022 +1000

    Get make lint passing

commit 611db3a
Author: Paul Hauner <[email protected]>
Date:   Fri Jun 17 14:57:08 2022 +1000

    Split head and FC locks

commit 7a44522
Author: Paul Hauner <[email protected]>
Date:   Thu Jun 16 18:22:32 2022 +1000

    Use fast head in notifier

commit 7199425
Author: Paul Hauner <[email protected]>
Date:   Thu Jun 16 18:19:08 2022 +1000

    Use fast head in http_api

commit 4ffdb41
Author: Paul Hauner <[email protected]>
Date:   Thu Jun 16 18:17:30 2022 +1000

    Do timeout lock for beacon state metrics

commit 1d67297
Author: Paul Hauner <[email protected]>
Date:   Thu Jun 16 18:05:18 2022 +1000

    Ensure FFG checkpoints are consistent with head

commit 7f2db8e
Author: Paul Hauner <[email protected]>
Date:   Thu Jun 16 17:40:33 2022 +1000

    Update fast head earlier

commit 1a4bb67
Author: Paul Hauner <[email protected]>
Date:   Thu Jun 16 16:32:47 2022 +1000

    Remove inconsistent head errors

commit 161c923
Author: Paul Hauner <[email protected]>
Date:   Thu Jun 16 15:36:27 2022 +1000

    Add fast canonical head

commit 888b278
Author: Paul Hauner <[email protected]>
Date:   Thu Jun 16 13:25:58 2022 +1000

    Avoid blocking async on canon head lock

commit 7552f9a
Author: Paul Hauner <[email protected]>
Date:   Wed Jun 15 19:19:09 2022 +1000

    Use state roots iter directly

commit 5b2f46a
Author: Paul Hauner <[email protected]>
Date:   Wed Jun 15 19:06:42 2022 +1000

    Compute state root in spawned function

commit 1603b44
Author: Paul Hauner <[email protected]>
Date:   Wed Jun 15 19:05:55 2022 +1000

    Drop txn_lock

commit 6426347
Author: Paul Hauner <[email protected]>
Date:   Wed Jun 15 17:07:42 2022 +1000

    Add weight logging

commit 52cb170
Author: Paul Hauner <[email protected]>
Date:   Wed Jun 15 16:31:55 2022 +1000

    Revert "Remove debug formatting for roots"

    This reverts commit 3f141ff.

commit 98bed99
Author: Paul Hauner <[email protected]>
Date:   Wed Jun 15 16:31:37 2022 +1000

    Raise gossip block log levels

commit 4badadf
Author: Paul Hauner <[email protected]>
Date:   Wed Jun 15 16:17:45 2022 +1000

    Remove debug formatting for roots

commit 0ab1b64
Author: Paul Hauner <[email protected]>
Date:   Wed Jun 15 15:54:44 2022 +1000

    Add back warning for opt head

commit d84550f
Author: Paul Hauner <[email protected]>
Date:   Wed Jun 15 15:08:23 2022 +1000

    Don't log crit when tokio stops block import

commit 90730a7
Author: Paul Hauner <[email protected]>
Date:   Wed Jun 15 14:47:32 2022 +1000

    Provide different state root to db migrator

commit be38daa
Author: Paul Hauner <[email protected]>
Date:   Wed Jun 15 14:28:48 2022 +1000

    Spawn new task for after_finalization

commit ccdc74d
Author: Paul Hauner <[email protected]>
Date:   Wed Jun 15 14:17:54 2022 +1000

    Arc-ify BeaconSnapshot

commit 67ca5d7
Author: Paul Hauner <[email protected]>
Date:   Wed Jun 15 09:59:04 2022 +1000

    Remove head_proposer_shuffling_decision_root

commit 5cda3ee
Author: Paul Hauner <[email protected]>
Date:   Wed Jun 15 09:52:53 2022 +1000

    Move recompute_head into canonical_head

commit 1a3070f
Author: Paul Hauner <[email protected]>
Date:   Tue Jun 14 18:58:29 2022 +1000

    Patch and unify status_message

commit 8f24eb1
Author: Paul Hauner <[email protected]>
Date:   Tue Jun 14 16:48:41 2022 +1000

    Make block_on_dangerous respect exit, metrics

commit de8482d
Author: Paul Hauner <[email protected]>
Date:   Tue Jun 14 16:31:07 2022 +1000

    Simplify getting for active val count

commit 67fd2a4
Author: Paul Hauner <[email protected]>
Date:   Tue Jun 14 16:07:04 2022 +1000

    Disable rayon for fork choice tests

commit 706e385
Author: Paul Hauner <[email protected]>
Date:   Sat Jun 11 16:54:01 2022 +1000

    Remove block_on for blinded blocks in HTTP API

commit cb8677c
Author: Paul Hauner <[email protected]>
Date:   Sat Jun 11 16:41:39 2022 +1000

    Remove `block_on` from EL

commit 81d7fc7
Author: Paul Hauner <[email protected]>
Date:   Sat Jun 11 16:23:52 2022 +1000

    Remove genesis alias workaround from EF tests

commit 52bbd94
Author: Paul Hauner <[email protected]>
Date:   Sat Jun 11 10:36:16 2022 +1000

    Fix compile error in client builder

commit 1faf17a
Author: Paul Hauner <[email protected]>
Date:   Sat Jun 11 10:34:35 2022 +1000

    Fix http_api compile error

commit ab4e1e8
Author: Paul Hauner <[email protected]>
Date:   Fri Jun 10 16:33:39 2022 +1000

    Move EL update spawning

commit 93864b8
Author: Paul Hauner <[email protected]>
Date:   Fri Jun 10 15:03:26 2022 +1000

    Fix weak subjectivity test

commit a861777
Author: Paul Hauner <[email protected]>
Date:   Fri Jun 10 14:37:13 2022 +1000

    Remove commented-out code

commit a6a6aae
Author: Paul Hauner <[email protected]>
Date:   Fri Jun 10 14:37:01 2022 +1000

    Fix store test

commit 2296643
Author: Paul Hauner <[email protected]>
Date:   Fri Jun 10 13:47:31 2022 +1000

    Fix failing payload_invalidation tests

commit 7b7b356
Author: Paul Hauner <[email protected]>
Date:   Wed Jun 8 18:20:02 2022 +1000

    Fix missing `!`

commit 1bcf234
Author: Paul Hauner <[email protected]>
Date:   Wed Jun 8 16:44:52 2022 +1000

    Remove some blocking methods

commit 5be6374
Author: Paul Hauner <[email protected]>
Date:   Wed Jun 8 13:48:09 2022 +1000

    Simplify chain segment processing

commit 2e9a8c0
Author: Paul Hauner <[email protected]>
Date:   Wed Jun 8 09:53:44 2022 +1000

    Fix compile errors in beacon chain tests

commit b7bee40
Author: Paul Hauner <[email protected]>
Date:   Tue Jun 7 17:22:55 2022 +1000

    Remove unused error

commit c3d04bd
Author: Paul Hauner <[email protected]>
Date:   Tue Jun 7 17:22:08 2022 +1000

    Remove HeadSafetyStatus

commit d305f49
Author: Paul Hauner <[email protected]>
Date:   Tue Jun 7 16:36:26 2022 +1000

    Fix failing network tests

commit 11a3eaa
Author: Paul Hauner <[email protected]>
Date:   Tue Jun 7 15:11:11 2022 +1000

    Fix MSRV issue

commit 49fce95
Author: Paul Hauner <[email protected]>
Date:   Tue Jun 7 15:07:29 2022 +1000

    Update test for finalized_checkpoint event

commit f452bd8
Author: Paul Hauner <[email protected]>
Date:   Tue Jun 7 11:53:20 2022 +1000

    Fix clippy lint

commit d21536e
Author: Paul Hauner <[email protected]>
Date:   Tue Jun 7 11:45:16 2022 +1000

    Remove deadlock from block import

commit dc4c7b6
Author: Paul Hauner <[email protected]>
Date:   Tue Jun 7 09:38:14 2022 +1000

    Fix justified checkpoint getter

commit f9634a6
Author: Paul Hauner <[email protected]>
Date:   Mon Jun 6 16:08:58 2022 +1000

    Fix test compilation errors

commit 59351d2
Author: Paul Hauner <[email protected]>
Date:   Mon Jun 6 15:43:12 2022 +1000

    Make block production async (make lint passing)

commit 689f3e6
Author: Paul Hauner <[email protected]>
Date:   Mon Jun 6 13:54:50 2022 +1000

    Refactor exec payload "prepare payload"

commit 3e21adb
Author: Paul Hauner <[email protected]>
Date:   Mon Jun 6 10:55:21 2022 +1000

    Fix http_api, `make lint` passes

commit 0a6b9ae
Author: Paul Hauner <[email protected]>
Date:   Mon Jun 6 10:42:43 2022 +1000

    Arc-ification of networking crate

commit 4597d24
Author: Paul Hauner <[email protected]>
Date:   Sun Jun 5 10:01:14 2022 +1000

    Start Arc-ifying SignedBeaconBlock

commit 2d0f5b9
Author: Paul Hauner <[email protected]>
Date:   Sun Jun 5 07:51:11 2022 +1000

    Partially add `PayloadNotifier`

commit ac35a43
Author: Paul Hauner <[email protected]>
Date:   Sat Jun 4 14:11:31 2022 +1000

    Reinstate panda banner

commit 3cbffa4
Author: Paul Hauner <[email protected]>
Date:   Thu Jun 2 18:53:51 2022 +1000

    Solve `StoreOp` lifetime problem

commit 87b4760
Author: Paul Hauner <[email protected]>
Date:   Thu Jun 2 18:43:51 2022 +1000

    Add progress on async block import

commit 9aa796f
Author: Paul Hauner <[email protected]>
Date:   Wed Jun 1 15:14:54 2022 +1000

    Make tests *finally* compile with async FC

commit 3438ac2
Author: Paul Hauner <[email protected]>
Date:   Tue May 31 20:22:42 2022 +1000

    More test fixes

commit 211e4cc
Author: Paul Hauner <[email protected]>
Date:   Tue May 31 18:04:57 2022 +1000

    Start *horrific* async test refactor

commit 095ff72
Author: Paul Hauner <[email protected]>
Date:   Tue May 31 16:39:36 2022 +1000

    Wrap harness in Arc

commit b00c0db
Author: Paul Hauner <[email protected]>
Date:   Tue May 31 15:54:51 2022 +1000

    Fix compile errors in HTTP tests

commit 8ade87e
Author: Paul Hauner <[email protected]>
Date:   Mon May 30 18:22:16 2022 +1000

    Remove old fork choice functions

commit ac01a7e
Author: Paul Hauner <[email protected]>
Date:   Tue Jun 7 11:51:22 2022 +1000

    Fix `http_api` to pass `make lint`

commit 4fa5be6
Author: Paul Hauner <[email protected]>
Date:   Mon May 30 13:34:10 2022 +1000

    Revert `common/eth2` to `unstable`

commit 61ff3b5
Author: Paul Hauner <[email protected]>
Date:   Mon May 30 13:30:27 2022 +1000

    Resolve `BeaconChain` compile errors

commit ec5f058
Author: Mac L <[email protected]>
Date:   Thu Mar 10 16:24:26 2022 +1100

    Squash previous works

    Update HTTP API to include `execution_optimistic` flag in responses

    Add `execution_optimistic` to server-sent events

    Fix formatting

    Fix tests

    Fix endpoint comment

    Remove redundent calls to `get_block`

    Fix route

    Return correct `execution_optimistic` values for api queries by state

    Updated api client functions

    Refactor HTTP API tests

    Return 404 when requested root is 0x000..000

    Tidy up

    Avoid recomputing head

    More improvements

    Use head when computing status for slot

    Squash several commit for 3175

    Start adding ChainView

    Ripple `ChainView` across code-base

    Fix after rebase on sigp#3070

    Pass `make lint` with `chain_view`

    Remove uses of get_head

    Remove uses of `with_head`

    Remove use of canonical head

    Start combining head and fork choice

    Big refactor to further lock joining

    Fix to pass `make lint`

    Fix consensus/fork_choice tests

    Fix http_api tests

    Remove current_slot from FC rebuild

    Fix lint issues after rebase

    Refactor out into recompute_head.rs

    Add TaskExecutor to BeaconChainHarness

    Thread task_executor into BeaconChain

    Use async fork choice

    Fix test compile errors

    Clear early attester cache sooner

    Move canonical_head to own file, fix test deadlock

    Get `make lint` working again

    Add `genesis_time` to `BeaconChain`

    Tidy

    Fix test compile issues

    Move recompute_head fns together

    Spawn EL update tasks

    Move EL updates into own function

    Tidy

commit 7d8f85f
Author: realbigsean <[email protected]>
Date:   Tue Jun 21 10:29:40 2022 -0600

    update justification and finalization method

commit a7b1e82
Merge: 2421281 c8315e1
Author: realbigsean <[email protected]>
Date:   Tue Jun 21 10:28:36 2022 -0600

    Merge branch 'fc-refactor' of https://github.com/realbigsean/lighthouse into fc-refactor

commit 2421281
Author: realbigsean <[email protected]>
Date:   Mon Jun 20 19:29:44 2022 -0400

    process justifiable for base as well

commit c8315e1
Author: realbigsean <[email protected]>
Date:   Mon Jun 20 13:49:21 2022 -0400

     EthSpec generic in proto array

commit 4ce3e40
Author: realbigsean <[email protected]>
Date:   Mon Jun 20 13:39:01 2022 -0400

    - pass thrugh spec argument
    - parse discard_equivocations test

commit 5b43461
Author: sean <[email protected]>
Date:   Sat Jun 18 13:02:39 2022 +0000

    remove ef-test proposer boost shim

commit 76b5344
Author: sean <[email protected]>
Date:   Sat Jun 18 12:16:08 2022 +0000

    revert changes to ef test tester

commit 3a0fea7
Author: sean <[email protected]>
Date:   Fri Jun 17 15:56:28 2022 +0000

    test

commit 0ce2a6b
Merge: 58b2892 7aeb9f9
Author: sean <[email protected]>
Date:   Fri Jun 17 15:18:16 2022 +0000

    Merge branch 'unstable' of https://github.com/sigp/lighthouse into fc-refactor

commit 58b2892
Author: sean <[email protected]>
Date:   Fri Jun 17 15:17:59 2022 +0000

    changes for tests

commit da9047c
Author: realbigsean <[email protected]>
Date:   Wed Jun 15 14:54:14 2022 -0400

    filter block tree changes

commit dc51a24
Author: realbigsean <[email protected]>
Date:   Tue Jun 14 19:24:49 2022 -0400

    updates for fc store unrealized justification tracking

commit e166d55
Merge: 8a5d9f3 564d7da
Author: realbigsean <[email protected]>
Date:   Tue Jun 14 16:00:52 2022 -0400

    Merge branch 'unstable' of https://github.com/sigp/lighthouse into fc-refactor

    # Conflicts:
    #	beacon_node/beacon_chain/src/schema_change.rs
    #	beacon_node/beacon_chain/src/schema_change/migration_schema_v9.rs
    #	beacon_node/http_api/src/attestation_performance.rs
    #	consensus/types/src/beacon_state/participation_cache.rs

commit 8a5d9f3
Author: realbigsean <[email protected]>
Date:   Thu May 12 09:04:29 2022 -0400

    Add scenario for n  + 2

commit 2ae43cd
Author: realbigsean <[email protected]>
Date:   Mon May 9 17:30:02 2022 -0400

    adjust pauls test to only run for altair and to, and to assert the head remains unchanged

commit 365d58c
Author: Paul Hauner <[email protected]>
Date:   Wed Apr 20 16:30:46 2022 +1000

    Add first unit test

commit 1854a23
Author: realbigsean <[email protected]>
Date:   Mon May 9 17:00:13 2022 -0400

    remove debugging lines, fix chache initialization slot

commit e67f07d
Author: realbigsean <[email protected]>
Date:   Mon May 9 10:06:56 2022 -0400

    debugging

commit 1b81c56
Author: realbigsean <[email protected]>
Date:   Fri May 6 11:57:51 2022 -0400

    fixes and debugging

commit 513d2b2
Author: realbigsean <[email protected]>
Date:   Thu May 5 18:21:13 2022 -0400

    refactors

commit 24c6273
Author: realbigsean <[email protected]>
Date:   Thu May 5 16:19:50 2022 -0400

    cargo fmt and fix

commit 24a5c16
Author: realbigsean <[email protected]>
Date:   Thu May 5 16:12:59 2022 -0400

    update filter block tree

commit fb16cd7
Author: realbigsean <[email protected]>
Date:   Thu May 5 10:04:23 2022 -0400

    fork choice schema migration

commit 6c7ab85
Author: realbigsean <[email protected]>
Date:   Tue May 3 08:12:27 2022 -0400

    Mutate state via mini beacon state in epoch processing

commit 5cae2f9
Author: realbigsean <[email protected]>
Date:   Sat Apr 23 09:04:33 2022 +0200

    separate participation cache by epoch and add it to the beacon state
@michaelsproul michaelsproul mentioned this pull request Jul 25, 2022
Closed
5 tasks
michaelsproul
michaelsproul approved these changes Jul 25, 2022
View reviewed changes
Copy link
Member

@michaelsproul michaelsproul left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

beacon_node/beacon_chain/src/schema_change/migration_schema_v7.rs
Comment on lines +101 to +103
// Need to downgrade the SSZ container to V7 so that all migrations can be applied in sequence.
let ssz_container = SszContainer::from(&fork_choice);
let ssz_container_v7 = SszContainerV7::from(ssz_container);
Copy link
Member

@michaelsproul michaelsproul Jul 25, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mm, this is a slightly nasty hack, but I think we should just leave it because we can delete all these migrations after the merge.

I'm noting this mostly for my own interest, I don't think we have any more schema migrations touching the ProtoNode, so this code should stand until we delete it

@realbigsean
Copy link
Member Author

bors r+

bors bot pushed a commit that referenced this pull request Jul 25, 2022
@realbigsean @paulhauner @macladson
Realized unrealized experimentation (#3322)
280e2e2 
## Issue Addressed

Add a flag that optionally enables unrealized vote tracking.  Would like to test out on testnets and benchmark differences in methods of vote tracking. This PR includes a DB schema upgrade to enable to new vote tracking style.


Co-authored-by: realbigsean <[email protected]>
Co-authored-by: Paul Hauner <[email protected]>
Co-authored-by: sean <[email protected]>
Co-authored-by: Mac L <[email protected]>
@bors
Copy link

bors bot commented Jul 25, 2022

Build failed:

@michaelsproul
Copy link
Member

bors retry

bors bot pushed a commit that referenced this pull request Jul 25, 2022
@realbigsean @paulhauner @macladson
Realized unrealized experimentation (#3322)
20ebf1f 
## Issue Addressed

Add a flag that optionally enables unrealized vote tracking.  Would like to test out on testnets and benchmark differences in methods of vote tracking. This PR includes a DB schema upgrade to enable to new vote tracking style.


Co-authored-by: realbigsean <[email protected]>
Co-authored-by: Paul Hauner <[email protected]>
Co-authored-by: sean <[email protected]>
Co-authored-by: Mac L <[email protected]>
@bors
Copy link

bors bot commented Jul 26, 2022

Pull request successfully merged into unstable.

Build succeeded:

@bors bors bot changed the title Realized unrealized experimentation [Merged by Bors] - Realized unrealized experimentation Jul 26, 2022
@bors bors bot closed this Jul 26, 2022
bors bot pushed a commit that referenced this pull request Jul 28, 2022
@michaelsproul
Remove equivocating validators from fork choice (#3371)
dfe34ad 
## Issue Addressed

Closes #3241
Closes #3242

## Proposed Changes

* [x] Implement logic to remove equivocating validators from fork choice per ethereum/consensus-specs#2845
* [x] Update tests to v1.2.0-rc.1. The new test which exercises `equivocating_indices` is passing.
* [x] Pull in some SSZ abstractions from the `tree-states` branch that make implementing Vec-compatible encoding for types like `BTreeSet` and `BTreeMap`.
* [x] Implement schema upgrades and downgrades for the database (new schema version is V11).
* [x] Apply attester slashings from blocks to fork choice

## Additional Info

* This PR doesn't need the `BTreeMap` impl, but `tree-states` does, and I don't think there's any harm in keeping it. But I could also be convinced to drop it.

Blocked on #3322.
bors bot pushed a commit that referenced this pull request Jul 28, 2022
@michaelsproul
Remove equivocating validators from fork choice (#3371)
a8382ae 
## Issue Addressed

Closes #3241
Closes #3242

## Proposed Changes

* [x] Implement logic to remove equivocating validators from fork choice per ethereum/consensus-specs#2845
* [x] Update tests to v1.2.0-rc.1. The new test which exercises `equivocating_indices` is passing.
* [x] Pull in some SSZ abstractions from the `tree-states` branch that make implementing Vec-compatible encoding for types like `BTreeSet` and `BTreeMap`.
* [x] Implement schema upgrades and downgrades for the database (new schema version is V11).
* [x] Apply attester slashings from blocks to fork choice

## Additional Info

* This PR doesn't need the `BTreeMap` impl, but `tree-states` does, and I don't think there's any harm in keeping it. But I could also be convinced to drop it.

Blocked on #3322.
bors bot pushed a commit that referenced this pull request Jul 28, 2022
@michaelsproul
Remove equivocating validators from fork choice (#3371)
c598945 
## Issue Addressed

Closes #3241
Closes #3242

## Proposed Changes

* [x] Implement logic to remove equivocating validators from fork choice per ethereum/consensus-specs#2845
* [x] Update tests to v1.2.0-rc.1. The new test which exercises `equivocating_indices` is passing.
* [x] Pull in some SSZ abstractions from the `tree-states` branch that make implementing Vec-compatible encoding for types like `BTreeSet` and `BTreeMap`.
* [x] Implement schema upgrades and downgrades for the database (new schema version is V11).
* [x] Apply attester slashings from blocks to fork choice

## Additional Info

* This PR doesn't need the `BTreeMap` impl, but `tree-states` does, and I don't think there's any harm in keeping it. But I could also be convinced to drop it.

Blocked on #3322.
bors bot pushed a commit that referenced this pull request Jul 28, 2022
@michaelsproul
Remove equivocating validators from fork choice (#3371)
1d52735 
## Issue Addressed

Closes #3241
Closes #3242

## Proposed Changes

* [x] Implement logic to remove equivocating validators from fork choice per ethereum/consensus-specs#2845
* [x] Update tests to v1.2.0-rc.1. The new test which exercises `equivocating_indices` is passing.
* [x] Pull in some SSZ abstractions from the `tree-states` branch that make implementing Vec-compatible encoding for types like `BTreeSet` and `BTreeMap`.
* [x] Implement schema upgrades and downgrades for the database (new schema version is V11).
* [x] Apply attester slashings from blocks to fork choice

## Additional Info

* This PR doesn't need the `BTreeMap` impl, but `tree-states` does, and I don't think there's any harm in keeping it. But I could also be convinced to drop it.

Blocked on #3322.
bors bot pushed a commit that referenced this pull request Jul 28, 2022
@michaelsproul
Remove equivocating validators from fork choice (#3371)
64522e9 
## Issue Addressed

Closes #3241
Closes #3242

## Proposed Changes

* [x] Implement logic to remove equivocating validators from fork choice per ethereum/consensus-specs#2845
* [x] Update tests to v1.2.0-rc.1. The new test which exercises `equivocating_indices` is passing.
* [x] Pull in some SSZ abstractions from the `tree-states` branch that make implementing Vec-compatible encoding for types like `BTreeSet` and `BTreeMap`.
* [x] Implement schema upgrades and downgrades for the database (new schema version is V11).
* [x] Apply attester slashings from blocks to fork choice

## Additional Info

* This PR doesn't need the `BTreeMap` impl, but `tree-states` does, and I don't think there's any harm in keeping it. But I could also be convinced to drop it.

Blocked on #3322.
bors bot pushed a commit that referenced this pull request Jul 28, 2022
@michaelsproul
Remove equivocating validators from fork choice (#3371)
d04fde3 
## Issue Addressed

Closes #3241
Closes #3242

## Proposed Changes

* [x] Implement logic to remove equivocating validators from fork choice per ethereum/consensus-specs#2845
* [x] Update tests to v1.2.0-rc.1. The new test which exercises `equivocating_indices` is passing.
* [x] Pull in some SSZ abstractions from the `tree-states` branch that make implementing Vec-compatible encoding for types like `BTreeSet` and `BTreeMap`.
* [x] Implement schema upgrades and downgrades for the database (new schema version is V11).
* [x] Apply attester slashings from blocks to fork choice

## Additional Info

* This PR doesn't need the `BTreeMap` impl, but `tree-states` does, and I don't think there's any harm in keeping it. But I could also be convinced to drop it.

Blocked on #3322.
@michaelsproul michaelsproul mentioned this pull request Jul 29, 2022
Closed
bors bot pushed a commit that referenced this pull request Jul 29, 2022
@michaelsproul
Enable count-unrealized by default (#3389)
33d1047 
## Issue Addressed

Enable #3322 by default on all networks.

The feature can be opted out of using `--count-unrealized=false` (the CLI flag is updated to take a parameter).
bors bot pushed a commit that referenced this pull request Jul 30, 2022
@michaelsproul
Enable count-unrealized by default (#3389)
fdfdb9b 
## Issue Addressed

Enable #3322 by default on all networks.

The feature can be opted out of using `--count-unrealized=false` (the CLI flag is updated to take a parameter).
@realbigsean realbigsean deleted the realized-unrealized-experimentation branch November 21, 2023 16:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@paulhauner paulhauner paulhauner approved these changes

@michaelsproul michaelsproul michaelsproul approved these changes

Assignees
No one assigned
Labels
backwards-incompat Backwards-incompatible API change ready-for-merge This PR is ready to merge. v2.5.0 Required for Goerli merge release
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@realbigsean @paulhauner @michaelsproul @macladson