[Merged by Bors] - Altair validator client and HTTP API #2404

michaelsproul · 2021-06-10T06:55:08Z

Proposed Changes

Implement the validator client and HTTP API changes necessary to support Altair

michaelsproul · 2021-06-10T06:59:13Z

This is based on #2302. The main commit is currently e1c40b5, I also made some changes to the testnet scripts in d1d2e2a.

Currently this will run with some subscription issues. It also falls over on skipped slots, and I've seen the occasional bad SyncAggregate signature on a period boundary (to be investigated).

To run the local testnet requires the minimal spec, i.e.

cargo install --path lighthouse --features spec-minimal --locked

michaelsproul · 2021-06-10T07:06:51Z

Currently I'm also just subscribing all committee members to subnets that they're in, regardless of whether or not they're known aggregators. I haven't yet thought deeply about whether we should change this (I suspect it's fine).

validator_client/src/duties_service/sync.rs

* Add v2 messages to rpc decoder * Ugly hack * Pass chainspec and genesis_root to Rpc * Add context bytes while encoding * Add a ForkContext struct * Pass ForkContext to rpc * crate compiles * Extract ForkContext into separate file; add a current_fork field * Fix encoding/decoding * Fix tests * Remove fork_schedule from rebase * Fix ForkContext * Fix tests * Remove fork_schedule again * Add altair empty and full block limits * Fix panic in snappy decoding * Fix limits * Move wrapping of RPCRequests to handler * RpcRequestContainer only used in OutboundUpgrade * Add altair blocks in rpc end to end tests * same rpc limits for V1 and V2 * V2 response decoding happens only for valid protocols * Add snappy response decoding tests * Add more snappy tests * Minor fixes * Appease clippy * to_context_bytes returns an Option * Add padding snappy message test for v2 * Minor fixes; remove accidentally added file * lint

@pawanjay176

* small refactors * Update attestation/deposit processing * Implement `upgrade_to_altair`, genesis init test * fix slashings and implement sync committee processing * refactor common epoch processing methods * delete unused imports * Fix typo in sync_committee_updates * Update to v1.1.0-alpha.3 * Fix remaining EF tests/handlers! * Fix beacon chain tests non-altair (#2305) * Use RefMut * Reset Cargo.lock to "stable" style * Tidy block tree hash * Temporarily hardcode base fork in op pool * Update block proposal & op pool * Fix bounds check on compute_sync_committee_indices * Tidy phase0 op pool * Remove global fork schedule (#2314) * Remove global fork schedule * Fix remaining EF tests for refactor * Get top-level compiling again * Bubble YAML config changes * Use superstruct partial getters This makes our Altair consensus logic compatible with future forks. * Lots of test refactors (#2315) * Fix beacon chain tests non-altair * fix op pool tests * fix merge * get block processing tests working * Refactor state transition vector tests, half are passing * cargo fmt * PR updates * Skip ahead to epoch 256 on beacon harness tests. Fix op pool tests. * clippy updates * More test cleanups and fixes * Use `upgrade_to_altair` in state transition * More test fixes * Begin adding pools related to sync committees * Add fork upgrade test, fix EF spec test * Sync committee pool refactors, attempt to get ssz tests working * Revert BitVector change * Update lcli and local testnet scripts * Fix `SyncCommitteeContribution` bit vector length, fix `SyncCommitteeSignature` field name * update docs and add tests for sync committee operation pool * More test updates, release-tests might pass now... * Nuke old store benchmark * Fix arbitrary instances * Participation bits fixes, clippy fixes * add BitVector intersection and union tests * Disable indexing in types & state processing * Delete tree hash and state proc benchmarks * Disable aggressive lints in tests * Appease udeps * Fix the RPC tests without hacks Thanks @pawanjay176 * Fix double update in github cargo fmt * Use published superstruct * Try splitting beacon_chain tests out of CI * Fix cargo-fmt --all * Abstract over fork in beacon_chain tests Also consolidate persistence_tests into store tests to reduce the # of integration test binaries slightly. * Fix clippy * Maintain backwards compatibility with std config * Sync committee signature verification work * Add an `Arc` to the `current_sync_committee` field, refactor `sync_committee_verification` code * Clean up imports * Move `sync_committee_base_epoch` to `Epoch` * Fmt and fix some lints * Add tests for new observation pools * Some op pool work * Provide clarity about BeaconState leaf count (#2339) * Remove sync committee indices cache We could add back a cache at some point, but it would likely look completely different due to having to compute the indices "backwards" from the public keys. Dealing with all the cases involving rebuilding the cache efficiently, or what to do if it isn't built will be quite involved so I think it's better we wait to see if we need it before optimising prematurely. * - Add beacon harness methods for sync signature create - separate sync signature `SubnetId` and `SelecitonProof` types from attestations - update op pool persistence to include sync aggregates * merge with `altair` * Add tests for decoding state/block with bad slot (#2341) * Tidy SyncCommittee types * More sync committee type cleanups/fallout * merge with `altair` * Some sync verification fixes and test progress * Add hack to allow harness with altair spec * Add test for applying base block to altair chain * Use graffiti len more consistently (#2348) * Replace a test deleted during Altair update (#2351) * Apply comment fixes from Paul Co-authored-by: Paul Hauner <[email protected]> * Add fork_name methods * Add fork checks and tests * Add tests for applying altair to base * Allow providing a custom spec to test harness * Fix compile error in network * Add tests for slot, epoch processing * Fix clippy lints * Fix release test * Finish op pool sync committee logic, add a couple tests. * Avoid slicing in PartialBeaconState (#2361) * [Altair] Make anonymous var-length SSZ decoding explicit (#2362) * Add explicit function for anon var length items * Remove unused import * Update comment * Use macros for naive aggregation pool tests. * small updates * [Altair] Rewards tests (#2349) * Rewards tests (base-only so far) * Altair rewards tests, clean-ups * Bump Rust version in Dockerfile * [Altair] Remove duplicate vars, panic on `u64::pow` overflow (#2350) * [Altair] Applying Base block to Altair chain (#2353) * Add hack to allow harness with altair spec * Add test for applying base block to altair chain * Add fork_name methods * Add fork checks and tests * Add tests for applying altair to base * Allow providing a custom spec to test harness * Fix compile error in network * Add tests for slot, epoch processing * Fix clippy lints * Fix release test * Fix test compile error * Fix test with inconsistent spec * Use fork_name_at_slot in genesis * [Altair] Add check for missed EF tests (#2346) * Rewards tests (base-only so far) * Altair rewards tests, clean-ups * Bump Rust version in Dockerfile * Add file access logging * Tidy * Fix clippy lint * Fix typo * Tidy, remove commented-out test exceptions * Improve naming in python script Co-authored-by: Michael Sproul <[email protected]> * Fix beacon chain harness bug in `make_sync_contributions`. Key the `ObservedSyncAggregators` pool on slot + subcommittee index * Accidentally removed a couple constants while merging * fix a typo in a const, and a bunch of lints * Restore lost CI YAML * Address review comments * Update comment * Return an error if total bal is 0 (#2366) * Fix and test selection proof `verify` * Fix get_sync_aggregate * update op pool tests to correctly use `get_sync_aggregate` * Minor changes to justification code (#2367) * [Altair] Optimization for `get_unslashed_participating_indices` (#2369) * Lift calculation of unslashed indices * Use HashSet for unslashed participating indices * Make Operation Pool Compile Again (MOPCA) * Add chainspec and genesis validators root to eth2_libp2p * Network altair rpc (#2301) * Add v2 messages to rpc decoder * Ugly hack * Pass chainspec and genesis_root to Rpc * Add context bytes while encoding * Add a ForkContext struct * Pass ForkContext to rpc * crate compiles * Extract ForkContext into separate file; add a current_fork field * Fix encoding/decoding * Fix tests * Remove fork_schedule from rebase * Fix ForkContext * Fix tests * Remove fork_schedule again * Add altair empty and full block limits * Fix panic in snappy decoding * Fix limits * Move wrapping of RPCRequests to handler * RpcRequestContainer only used in OutboundUpgrade * Add altair blocks in rpc end to end tests * same rpc limits for V1 and V2 * V2 response decoding happens only for valid protocols * Add snappy response decoding tests * Add more snappy tests * Minor fixes * Appease clippy * to_context_bytes returns an Option * Add padding snappy message test for v2 * Minor fixes; remove accidentally added file * lint * Add chainspec and genesis validators root to eth2_libp2p * Add altair gossip types * Subscribe to required gossip topics * Unsubscribe from pre fork topics * Compiles * Start adding sync committee attnets * Add Subnet enum to include SyncCommittee subnet_ids * update_enr_bitfield takes a Subnet instead of SubnetId * Fix network service * Add MetaData V2 * Don't read context bytes for MetaData V2 * Add sync committee topics to whitelist * Fix metadata encoding; fmt * Fix next_fork methods in chainspec * Minor fixes * Fix next_fork_update in network service; add next_unsubscribe delay * Accept peers which haven't updated next_fork_version/epoch * Add sync committee topic tests * Simplify required_gossip_digests; improve logging around fork boundary * Fix tests * Minor refactorings * Add sync contribution aggregation tests to the op pool and fix an op pool bug * Only advance state as necessary in sync committee verification tests. Fix compile error. * Delete `verify_sync_contributions.rs` and fix a comment. * Remove `Arc` from `current_sync_committee` * Add `max_capacity` to `ObservedAggregates` constructor * Update consensus code to v1.1.0-alpha.6 * Make the `SyncAggregateId` a fixed-length struct * Add metrics and update docs * Address review comments * Sync committee subnets * Ripple spec updates * Revert "Remove `Arc` from `current_sync_committee`" This reverts commit bf6324e. * fix some pool tests * Updates to some docs, general cleanup * Fix default capacity for sync committee observed caches * Fix default capacity for sync committee observed caches * Fix default capacity for sync committee observed caches * Fix more docs and metrics * Remove `is_valid` methods from `SignedAggregateAndProof` and `SignedContributionAndProof`. The logic is duplicated in the signature set verification methods. * remove outdated `FIXME` * Fix docs * Fix docs * Implement preset <> config distinction * Update confusing name `SyncCommitteeSubnetSize` -> `SyncSubcommitteeSize` * Updates for latest spec merge * update client for `lighthouse/spec` endpoint * Add back accidentally deleted import * Return ConfigAndPreset from VC API * Don't panic if there aren't any aggregators Rustfmt also returned with a vengeance to rearrange this block. * Check state root in transition tests * Fix clippy * Fix Pyrmont & Prater configs. Delete Toledo. * Add compatibility shim for /config/spec * Fix fork_digest in Status message * Revert "update client for `lighthouse/spec` endpoint" This reverts commit 7a9b2ac. # Conflicts: # validator_client/src/http_api/tests.rs * - Update `SYNC_COMMITTEE_SUBNET_COUNT` to alpha.6 - rename the `sync_aggregator_selection_data.rs` file - include sync committee ssz static tests in the `check_all_file_accessed.py` checks - Update `ObservedAggregates` consts so they are specific to the type of object being cached * - Update `NaiveAggregationPool`'s default capacity so it is specific to the type of object being inserted * fix `observed_aggregates` tests * Add chainspec and genesis validators root to eth2_libp2p * Network altair rpc (#2301) * Add v2 messages to rpc decoder * Ugly hack * Pass chainspec and genesis_root to Rpc * Add context bytes while encoding * Add a ForkContext struct * Pass ForkContext to rpc * crate compiles * Extract ForkContext into separate file; add a current_fork field * Fix encoding/decoding * Fix tests * Remove fork_schedule from rebase * Fix ForkContext * Fix tests * Remove fork_schedule again * Add altair empty and full block limits * Fix panic in snappy decoding * Fix limits * Move wrapping of RPCRequests to handler * RpcRequestContainer only used in OutboundUpgrade * Add altair blocks in rpc end to end tests * same rpc limits for V1 and V2 * V2 response decoding happens only for valid protocols * Add snappy response decoding tests * Add more snappy tests * Minor fixes * Appease clippy * to_context_bytes returns an Option * Add padding snappy message test for v2 * Minor fixes; remove accidentally added file * lint * Use SyncSubnetId for sync committee subnets * Add worker tasks for sync committee gossip messages * Use correct type level constants * Decode altair gossip blocks * Add a SyncCommitteeSubscribe network event * PR updates * fix op pool tests * Minor cleanups * Calculate `SyncAggregate` on demand * Address review comments * Op pool migration + superstruct * Fix op pool test * Update to spec v1.1.0-alpha.7 * Fix sync committee subscriptions * Add sync committee subscription test * Add chainspec and genesis validators root to eth2_libp2p * Network altair rpc (#2301) * Add v2 messages to rpc decoder * Ugly hack * Pass chainspec and genesis_root to Rpc * Add context bytes while encoding * Add a ForkContext struct * Pass ForkContext to rpc * crate compiles * Extract ForkContext into separate file; add a current_fork field * Fix encoding/decoding * Fix tests * Remove fork_schedule from rebase * Fix ForkContext * Fix tests * Remove fork_schedule again * Add altair empty and full block limits * Fix panic in snappy decoding * Fix limits * Move wrapping of RPCRequests to handler * RpcRequestContainer only used in OutboundUpgrade * Add altair blocks in rpc end to end tests * same rpc limits for V1 and V2 * V2 response decoding happens only for valid protocols * Add snappy response decoding tests * Add more snappy tests * Minor fixes * Appease clippy * to_context_bytes returns an Option * Add padding snappy message test for v2 * Minor fixes; remove accidentally added file * lint * - Fix sync committee verification on slot prior to sync committee period boundary - Add `Arc` to `next_sync_committee` - Other PR updates * Refactor how we retrieve `PublicKey` for sync signature verification * Key `observed_contributors` on `SlotSubcommitteeIndex` * Update mainnet preset YAML * Gossipsub message_id_fn changes * Add tests for sync committee verification at slot before sync committee period boundary * fix op pool tests * - Make `SyncSubnetId` a required argument for `verify_sync_signature_for_gossip` - Drop locks as soon as possible during metrics scrape * fix sync committee verification test macro * Fix beacon_chain test * rust 1.53.0 updates * add recursion limit to `simulator` crate * cargo fmt * `SyncCommitteSignature` -> `SyncCommitteeMessage` * Updates related to merge with `unstable` * More updates related to `SyncCommitteMessage` refactor * Couple more `unstable` merge fixes * Fix `SyncAggregate` field name * Add chainspec and genesis validators root to eth2_libp2p * Network altair rpc (#2301) * Add v2 messages to rpc decoder * Ugly hack * Pass chainspec and genesis_root to Rpc * Add context bytes while encoding * Add a ForkContext struct * Pass ForkContext to rpc * crate compiles * Extract ForkContext into separate file; add a current_fork field * Fix encoding/decoding * Fix tests * Remove fork_schedule from rebase * Fix ForkContext * Fix tests * Remove fork_schedule again * Add altair empty and full block limits * Fix panic in snappy decoding * Fix limits * Move wrapping of RPCRequests to handler * RpcRequestContainer only used in OutboundUpgrade * Add altair blocks in rpc end to end tests * same rpc limits for V1 and V2 * V2 response decoding happens only for valid protocols * Add snappy response decoding tests * Add more snappy tests * Minor fixes * Appease clippy * to_context_bytes returns an Option * Add padding snappy message test for v2 * Minor fixes; remove accidentally added file * lint * Change libp2p dependency for message_id_fn * appease clippy * Add sync committee related metrics * Expand and refactor fork functions * Simplify compute_subnets_for_sync_committee * Fix subnet service test * Add v2 protocols for OutboundRequests missed in the merge * Handle sync committee gossip errors * Remove delay for updating fork * fix the expected error in the sync contribution verification test at the sync period boundary * Updates related to merge with `unstable` * Couple more `unstable` merge fixes * Split rpc encode/decode into smaller functions * Import some relevant funcs from the VC Based on "Expand and refactor fork functions" * Fix issues with skip slots at fork boundaries * Fix clippy again * alpha.8 updates * Pop sync committee tasks from beacon processor * Fix some gossip scoring severity issue * Fix bug in peer_manager; Improve logging around subnet discoveries Co-authored-by: realbigsean <[email protected]> Co-authored-by: Michael Sproul <[email protected]> Co-authored-by: realbigsean <[email protected]> Co-authored-by: Paul Hauner <[email protected]>

michaelsproul · 2021-07-22T03:13:48Z

Rebased, as that seemed to be easiest way to clean up conflicts after network-altair rebased on unstable.

…alid

michaelsproul · 2021-07-26T08:33:04Z

Updated to only pre-compute selection proofs 2 epochs in advance, instead of for the whole sync committee period. This prevents CPU usage spikes on restart. Currently running on altair-devnet-1.

michaelsproul · 2021-08-03T08:05:09Z

Ready for review!

michaelsproul · 2021-08-04T01:54:18Z

I'm fixing the doppel tests, it just seems to be a CLI arg issue (SPEC_PRESET not defined)

@pawanjay176

commit 410d751 Merge: 8e7466a 6a6d175 Author: Michael Sproul <[email protected]> Date: Tue Aug 3 18:02:45 2021 +1000 Merge remote-tracking branch 'origin/network-altair' into altair-vc commit 8e7466a Author: Michael Sproul <[email protected]> Date: Tue Aug 3 18:02:21 2021 +1000 Fix beacon chain tests commit 6a6d175 Author: pawan <[email protected]> Date: Tue Aug 3 12:49:30 2021 +0530 Address more review comments commit 3266fcd Author: Michael Sproul <[email protected]> Date: Tue Aug 3 17:20:15 2021 +1000 Fix clippy commit 86b68a4 Merge: 84830a8 68357f3 Author: Michael Sproul <[email protected]> Date: Tue Aug 3 17:13:52 2021 +1000 Merge remote-tracking branch 'origin/network-altair' into altair-vc commit 84830a8 Author: Michael Sproul <[email protected]> Date: Tue Aug 3 16:53:51 2021 +1000 Implement API for sync committee indices commit 68357f3 Author: pawan <[email protected]> Date: Mon Aug 2 13:44:44 2021 +0530 Remove sync_committee_subnet_count from ChainSpec commit 940535c Merge: 05fe793 c5786a8 Author: pawan <[email protected]> Date: Mon Aug 2 13:09:04 2021 +0530 Merge branch 'unstable' into network-altair commit 05fe793 Author: pawan <[email protected]> Date: Mon Aug 2 12:07:49 2021 +0530 Fix issues from review commit 7b3a0a1 Author: Michael Sproul <[email protected]> Date: Mon Aug 2 16:11:02 2021 +1000 Tweak log levels commit 8d6cdf9 Merge: d1bbf1a c5786a8 Author: Michael Sproul <[email protected]> Date: Mon Aug 2 14:46:21 2021 +1000 Merge remote-tracking branch 'origin/unstable' into altair-vc commit d1bbf1a Author: Michael Sproul <[email protected]> Date: Fri Jul 30 12:46:26 2021 +1000 Add /v2/ API endpoints commit b9d134e Author: Michael Sproul <[email protected]> Date: Mon Jul 26 12:37:49 2021 +1000 Add target-peers and spec preset to local testnet commit 0c25e92 Author: Michael Sproul <[email protected]> Date: Mon Jul 26 12:34:34 2021 +1000 Prevent bootnode adding its own ENR commit a3fb7b3 Author: Michael Sproul <[email protected]> Date: Mon Jul 26 12:34:11 2021 +1000 Limit selection proof pre-computation commit d0e2153 Author: Michael Sproul <[email protected]> Date: Fri Jul 23 14:23:15 2021 +1000 Delete the fork service commit e63b5de Author: Michael Sproul <[email protected]> Date: Thu Jul 22 12:11:43 2021 +1000 Altair validator client and API commit 2b04ddf Merge: 59a32a3 923486f Author: pawan <[email protected]> Date: Wed Jul 28 14:45:57 2021 +0530 Merge branch 'unstable' into network-altair commit 59a32a3 Author: pawan <[email protected]> Date: Sat Jul 24 00:18:47 2021 +0530 Fix minor issues commit 0656056 Author: pawan <[email protected]> Date: Fri Jul 23 18:26:57 2021 +0530 Change libp2p dependency to sigp fork commit bed92c9 Author: realbigsean <[email protected]> Date: Tue Jul 13 12:21:56 2021 -0400 fix incorrect method name and update test to ensure off-by-one is invalid commit 6c76853 Merge: 2e35733 63923ea Author: pawan <[email protected]> Date: Wed Jul 21 23:31:30 2021 +0530 Merge discv5 upstream changes commit 2e35733 Author: pawan <[email protected]> Date: Tue Jul 20 16:14:34 2021 +0530 Maintain sync committee peers commit 9f29788 Author: pawan <[email protected]> Date: Mon Jul 19 15:42:12 2021 +0530 Fix subnet service test commit 9f9c184 Merge: 3d42f06 9a8320b Author: pawan <[email protected]> Date: Fri Jul 16 20:44:09 2021 +0530 Merge network-1.5 changes commit 3d42f06 Author: Pawan Dhananjay <[email protected]> Date: Fri Jul 16 18:56:38 2021 +0530 Network altair gossip (#2464) * small refactors * Update attestation/deposit processing * Implement `upgrade_to_altair`, genesis init test * fix slashings and implement sync committee processing * refactor common epoch processing methods * delete unused imports * Fix typo in sync_committee_updates * Update to v1.1.0-alpha.3 * Fix remaining EF tests/handlers! * Fix beacon chain tests non-altair (#2305) * Use RefMut * Reset Cargo.lock to "stable" style * Tidy block tree hash * Temporarily hardcode base fork in op pool * Update block proposal & op pool * Fix bounds check on compute_sync_committee_indices * Tidy phase0 op pool * Remove global fork schedule (#2314) * Remove global fork schedule * Fix remaining EF tests for refactor * Get top-level compiling again * Bubble YAML config changes * Use superstruct partial getters This makes our Altair consensus logic compatible with future forks. * Lots of test refactors (#2315) * Fix beacon chain tests non-altair * fix op pool tests * fix merge * get block processing tests working * Refactor state transition vector tests, half are passing * cargo fmt * PR updates * Skip ahead to epoch 256 on beacon harness tests. Fix op pool tests. * clippy updates * More test cleanups and fixes * Use `upgrade_to_altair` in state transition * More test fixes * Begin adding pools related to sync committees * Add fork upgrade test, fix EF spec test * Sync committee pool refactors, attempt to get ssz tests working * Revert BitVector change * Update lcli and local testnet scripts * Fix `SyncCommitteeContribution` bit vector length, fix `SyncCommitteeSignature` field name * update docs and add tests for sync committee operation pool * More test updates, release-tests might pass now... * Nuke old store benchmark * Fix arbitrary instances * Participation bits fixes, clippy fixes * add BitVector intersection and union tests * Disable indexing in types & state processing * Delete tree hash and state proc benchmarks * Disable aggressive lints in tests * Appease udeps * Fix the RPC tests without hacks Thanks @pawanjay176 * Fix double update in github cargo fmt * Use published superstruct * Try splitting beacon_chain tests out of CI * Fix cargo-fmt --all * Abstract over fork in beacon_chain tests Also consolidate persistence_tests into store tests to reduce the # of integration test binaries slightly. * Fix clippy * Maintain backwards compatibility with std config * Sync committee signature verification work * Add an `Arc` to the `current_sync_committee` field, refactor `sync_committee_verification` code * Clean up imports * Move `sync_committee_base_epoch` to `Epoch` * Fmt and fix some lints * Add tests for new observation pools * Some op pool work * Provide clarity about BeaconState leaf count (#2339) * Remove sync committee indices cache We could add back a cache at some point, but it would likely look completely different due to having to compute the indices "backwards" from the public keys. Dealing with all the cases involving rebuilding the cache efficiently, or what to do if it isn't built will be quite involved so I think it's better we wait to see if we need it before optimising prematurely. * - Add beacon harness methods for sync signature create - separate sync signature `SubnetId` and `SelecitonProof` types from attestations - update op pool persistence to include sync aggregates * merge with `altair` * Add tests for decoding state/block with bad slot (#2341) * Tidy SyncCommittee types * More sync committee type cleanups/fallout * merge with `altair` * Some sync verification fixes and test progress * Add hack to allow harness with altair spec * Add test for applying base block to altair chain * Use graffiti len more consistently (#2348) * Replace a test deleted during Altair update (#2351) * Apply comment fixes from Paul Co-authored-by: Paul Hauner <[email protected]> * Add fork_name methods * Add fork checks and tests * Add tests for applying altair to base * Allow providing a custom spec to test harness * Fix compile error in network * Add tests for slot, epoch processing * Fix clippy lints * Fix release test * Finish op pool sync committee logic, add a couple tests. * Avoid slicing in PartialBeaconState (#2361) * [Altair] Make anonymous var-length SSZ decoding explicit (#2362) * Add explicit function for anon var length items * Remove unused import * Update comment * Use macros for naive aggregation pool tests. * small updates * [Altair] Rewards tests (#2349) * Rewards tests (base-only so far) * Altair rewards tests, clean-ups * Bump Rust version in Dockerfile * [Altair] Remove duplicate vars, panic on `u64::pow` overflow (#2350) * [Altair] Applying Base block to Altair chain (#2353) * Add hack to allow harness with altair spec * Add test for applying base block to altair chain * Add fork_name methods * Add fork checks and tests * Add tests for applying altair to base * Allow providing a custom spec to test harness * Fix compile error in network * Add tests for slot, epoch processing * Fix clippy lints * Fix release test * Fix test compile error * Fix test with inconsistent spec * Use fork_name_at_slot in genesis * [Altair] Add check for missed EF tests (#2346) * Rewards tests (base-only so far) * Altair rewards tests, clean-ups * Bump Rust version in Dockerfile * Add file access logging * Tidy * Fix clippy lint * Fix typo * Tidy, remove commented-out test exceptions * Improve naming in python script Co-authored-by: Michael Sproul <[email protected]> * Fix beacon chain harness bug in `make_sync_contributions`. Key the `ObservedSyncAggregators` pool on slot + subcommittee index * Accidentally removed a couple constants while merging * fix a typo in a const, and a bunch of lints * Restore lost CI YAML * Address review comments * Update comment * Return an error if total bal is 0 (#2366) * Fix and test selection proof `verify` * Fix get_sync_aggregate * update op pool tests to correctly use `get_sync_aggregate` * Minor changes to justification code (#2367) * [Altair] Optimization for `get_unslashed_participating_indices` (#2369) * Lift calculation of unslashed indices * Use HashSet for unslashed participating indices * Make Operation Pool Compile Again (MOPCA) * Add chainspec and genesis validators root to eth2_libp2p * Network altair rpc (#2301) * Add v2 messages to rpc decoder * Ugly hack * Pass chainspec and genesis_root to Rpc * Add context bytes while encoding * Add a ForkContext struct * Pass ForkContext to rpc * crate compiles * Extract ForkContext into separate file; add a current_fork field * Fix encoding/decoding * Fix tests * Remove fork_schedule from rebase * Fix ForkContext * Fix tests * Remove fork_schedule again * Add altair empty and full block limits * Fix panic in snappy decoding * Fix limits * Move wrapping of RPCRequests to handler * RpcRequestContainer only used in OutboundUpgrade * Add altair blocks in rpc end to end tests * same rpc limits for V1 and V2 * V2 response decoding happens only for valid protocols * Add snappy response decoding tests * Add more snappy tests * Minor fixes * Appease clippy * to_context_bytes returns an Option * Add padding snappy message test for v2 * Minor fixes; remove accidentally added file * lint * Add chainspec and genesis validators root to eth2_libp2p * Add altair gossip types * Subscribe to required gossip topics * Unsubscribe from pre fork topics * Compiles * Start adding sync committee attnets * Add Subnet enum to include SyncCommittee subnet_ids * update_enr_bitfield takes a Subnet instead of SubnetId * Fix network service * Add MetaData V2 * Don't read context bytes for MetaData V2 * Add sync committee topics to whitelist * Fix metadata encoding; fmt * Fix next_fork methods in chainspec * Minor fixes * Fix next_fork_update in network service; add next_unsubscribe delay * Accept peers which haven't updated next_fork_version/epoch * Add sync committee topic tests * Simplify required_gossip_digests; improve logging around fork boundary * Fix tests * Minor refactorings * Add sync contribution aggregation tests to the op pool and fix an op pool bug * Only advance state as necessary in sync committee verification tests. Fix compile error. * Delete `verify_sync_contributions.rs` and fix a comment. * Remove `Arc` from `current_sync_committee` * Add `max_capacity` to `ObservedAggregates` constructor * Update consensus code to v1.1.0-alpha.6 * Make the `SyncAggregateId` a fixed-length struct * Add metrics and update docs * Address review comments * Sync committee subnets * Ripple spec updates * Revert "Remove `Arc` from `current_sync_committee`" This reverts commit bf6324e. * fix some pool tests * Updates to some docs, general cleanup * Fix default capacity for sync committee observed caches * Fix default capacity for sync committee observed caches * Fix default capacity for sync committee observed caches * Fix more docs and metrics * Remove `is_valid` methods from `SignedAggregateAndProof` and `SignedContributionAndProof`. The logic is duplicated in the signature set verification methods. * remove outdated `FIXME` * Fix docs * Fix docs * Implement preset <> config distinction * Update confusing name `SyncCommitteeSubnetSize` -> `SyncSubcommitteeSize` * Updates for latest spec merge * update client for `lighthouse/spec` endpoint * Add back accidentally deleted import * Return ConfigAndPreset from VC API * Don't panic if there aren't any aggregators Rustfmt also returned with a vengeance to rearrange this block. * Check state root in transition tests * Fix clippy * Fix Pyrmont & Prater configs. Delete Toledo. * Add compatibility shim for /config/spec * Fix fork_digest in Status message * Revert "update client for `lighthouse/spec` endpoint" This reverts commit 7a9b2ac. # Conflicts: # validator_client/src/http_api/tests.rs * - Update `SYNC_COMMITTEE_SUBNET_COUNT` to alpha.6 - rename the `sync_aggregator_selection_data.rs` file - include sync committee ssz static tests in the `check_all_file_accessed.py` checks - Update `ObservedAggregates` consts so they are specific to the type of object being cached * - Update `NaiveAggregationPool`'s default capacity so it is specific to the type of object being inserted * fix `observed_aggregates` tests * Add chainspec and genesis validators root to eth2_libp2p * Network altair rpc (#2301) * Add v2 messages to rpc decoder * Ugly hack * Pass chainspec and genesis_root to Rpc * Add context bytes while encoding * Add a ForkContext struct * Pass ForkContext to rpc * crate compiles * Extract ForkContext into separate file; add a current_fork field * Fix encoding/decoding * Fix tests * Remove fork_schedule from rebase * Fix ForkContext * Fix tests * Remove fork_schedule again * Add altair empty and full block limits * Fix panic in snappy decoding * Fix limits * Move wrapping of RPCRequests to handler * RpcRequestContainer only used in OutboundUpgrade * Add altair blocks in rpc end to end tests * same rpc limits for V1 and V2 * V2 response decoding happens only for valid protocols * Add snappy response decoding tests * Add more snappy tests * Minor fixes * Appease clippy * to_context_bytes returns an Option * Add padding snappy message test for v2 * Minor fixes; remove accidentally added file * lint * Use SyncSubnetId for sync committee subnets * Add worker tasks for sync committee gossip messages * Use correct type level constants * Decode altair gossip blocks * Add a SyncCommitteeSubscribe network event * PR updates * fix op pool tests * Minor cleanups * Calculate `SyncAggregate` on demand * Address review comments * Op pool migration + superstruct * Fix op pool test * Update to spec v1.1.0-alpha.7 * Fix sync committee subscriptions * Add sync committee subscription test * Add chainspec and genesis validators root to eth2_libp2p * Network altair rpc (#2301) * Add v2 messages to rpc decoder * Ugly hack * Pass chainspec and genesis_root to Rpc * Add context bytes while encoding * Add a ForkContext struct * Pass ForkContext to rpc * crate compiles * Extract ForkContext into separate file; add a current_fork field * Fix encoding/decoding * Fix tests * Remove fork_schedule from rebase * Fix ForkContext * Fix tests * Remove fork_schedule again * Add altair empty and full block limits * Fix panic in snappy decoding * Fix limits * Move wrapping of RPCRequests to handler * RpcRequestContainer only used in OutboundUpgrade * Add altair blocks in rpc end to end tests * same rpc limits for V1 and V2 * V2 response decoding happens only for valid protocols * Add snappy response decoding tests * Add more snappy tests * Minor fixes * Appease clippy * to_context_bytes returns an Option * Add padding snappy message test for v2 * Minor fixes; remove accidentally added file * lint * - Fix sync committee verification on slot prior to sync committee period boundary - Add `Arc` to `next_sync_committee` - Other PR updates * Refactor how we retrieve `PublicKey` for sync signature verification * Key `observed_contributors` on `SlotSubcommitteeIndex` * Update mainnet preset YAML * Gossipsub message_id_fn changes * Add tests for sync committee verification at slot before sync committee period boundary * fix op pool tests * - Make `SyncSubnetId` a required argument for `verify_sync_signature_for_gossip` - Drop locks as soon as possible during metrics scrape * fix sync committee verification test macro * Fix beacon_chain test * rust 1.53.0 updates * add recursion limit to `simulator` crate * cargo fmt * `SyncCommitteSignature` -> `SyncCommitteeMessage` * Updates related to merge with `unstable` * More updates related to `SyncCommitteMessage` refactor * Couple more `unstable` merge fixes * Fix `SyncAggregate` field name * Add chainspec and genesis validators root to eth2_libp2p * Network altair rpc (#2301) * Add v2 messages to rpc decoder * Ugly hack * Pass chainspec and genesis_root to Rpc * Add context bytes while encoding * Add a ForkContext struct * Pass ForkContext to rpc * crate compiles * Extract ForkContext into separate file; add a current_fork field * Fix encoding/decoding * Fix tests * Remove fork_schedule from rebase * Fix ForkContext * Fix tests * Remove fork_schedule again * Add altair empty and full block limits * Fix panic in snappy decoding * Fix limits * Move wrapping of RPCRequests to handler * RpcRequestContainer only used in OutboundUpgrade * Add altair blocks in rpc end to end tests * same rpc limits for V1 and V2 * V2 response decoding happens only for valid protocols * Add snappy response decoding tests * Add more snappy tests * Minor fixes * Appease clippy * to_context_bytes returns an Option * Add padding snappy message test for v2 * Minor fixes; remove accidentally added file * lint * Change libp2p dependency for message_id_fn * appease clippy * Add sync committee related metrics * Expand and refactor fork functions * Simplify compute_subnets_for_sync_committee * Fix subnet service test * Add v2 protocols for OutboundRequests missed in the merge * Handle sync committee gossip errors * Remove delay for updating fork * fix the expected error in the sync contribution verification test at the sync period boundary * Updates related to merge with `unstable` * Couple more `unstable` merge fixes * Split rpc encode/decode into smaller functions * Import some relevant funcs from the VC Based on "Expand and refactor fork functions" * Fix issues with skip slots at fork boundaries * Fix clippy again * alpha.8 updates * Pop sync committee tasks from beacon processor * Fix some gossip scoring severity issue * Fix bug in peer_manager; Improve logging around subnet discoveries Co-authored-by: realbigsean <[email protected]> Co-authored-by: Michael Sproul <[email protected]> Co-authored-by: realbigsean <[email protected]> Co-authored-by: Paul Hauner <[email protected]> commit f37f267 Author: Pawan Dhananjay <[email protected]> Date: Tue May 11 10:30:43 2021 +0530 Network altair rpc (#2301) * Add v2 messages to rpc decoder * Ugly hack * Pass chainspec and genesis_root to Rpc * Add context bytes while encoding * Add a ForkContext struct * Pass ForkContext to rpc * crate compiles * Extract ForkContext into separate file; add a current_fork field * Fix encoding/decoding * Fix tests * Remove fork_schedule from rebase * Fix ForkContext * Fix tests * Remove fork_schedule again * Add altair empty and full block limits * Fix panic in snappy decoding * Fix limits * Move wrapping of RPCRequests to handler * RpcRequestContainer only used in OutboundUpgrade * Add altair blocks in rpc end to end tests * same rpc limits for V1 and V2 * V2 response decoding happens only for valid protocols * Add snappy response decoding tests * Add more snappy tests * Minor fixes * Appease clippy * to_context_bytes returns an Option * Add padding snappy message test for v2 * Minor fixes; remove accidentally added file * lint commit b477bdd Author: pawan <[email protected]> Date: Mon Apr 5 19:30:48 2021 +0530 Add chainspec and genesis validators root to eth2_libp2p

@pawanjay176

commit 410d751 Merge: 8e7466a 6a6d175 Author: Michael Sproul <[email protected]> Date: Tue Aug 3 18:02:45 2021 +1000 Merge remote-tracking branch 'origin/network-altair' into altair-vc commit 8e7466a Author: Michael Sproul <[email protected]> Date: Tue Aug 3 18:02:21 2021 +1000 Fix beacon chain tests commit 6a6d175 Author: pawan <[email protected]> Date: Tue Aug 3 12:49:30 2021 +0530 Address more review comments commit 3266fcd Author: Michael Sproul <[email protected]> Date: Tue Aug 3 17:20:15 2021 +1000 Fix clippy commit 86b68a4 Merge: 84830a8 68357f3 Author: Michael Sproul <[email protected]> Date: Tue Aug 3 17:13:52 2021 +1000 Merge remote-tracking branch 'origin/network-altair' into altair-vc commit 84830a8 Author: Michael Sproul <[email protected]> Date: Tue Aug 3 16:53:51 2021 +1000 Implement API for sync committee indices commit 68357f3 Author: pawan <[email protected]> Date: Mon Aug 2 13:44:44 2021 +0530 Remove sync_committee_subnet_count from ChainSpec commit 940535c Merge: 05fe793 c5786a8 Author: pawan <[email protected]> Date: Mon Aug 2 13:09:04 2021 +0530 Merge branch 'unstable' into network-altair commit 05fe793 Author: pawan <[email protected]> Date: Mon Aug 2 12:07:49 2021 +0530 Fix issues from review commit 7b3a0a1 Author: Michael Sproul <[email protected]> Date: Mon Aug 2 16:11:02 2021 +1000 Tweak log levels commit 8d6cdf9 Merge: d1bbf1a c5786a8 Author: Michael Sproul <[email protected]> Date: Mon Aug 2 14:46:21 2021 +1000 Merge remote-tracking branch 'origin/unstable' into altair-vc commit d1bbf1a Author: Michael Sproul <[email protected]> Date: Fri Jul 30 12:46:26 2021 +1000 Add /v2/ API endpoints commit b9d134e Author: Michael Sproul <[email protected]> Date: Mon Jul 26 12:37:49 2021 +1000 Add target-peers and spec preset to local testnet commit 0c25e92 Author: Michael Sproul <[email protected]> Date: Mon Jul 26 12:34:34 2021 +1000 Prevent bootnode adding its own ENR commit a3fb7b3 Author: Michael Sproul <[email protected]> Date: Mon Jul 26 12:34:11 2021 +1000 Limit selection proof pre-computation commit d0e2153 Author: Michael Sproul <[email protected]> Date: Fri Jul 23 14:23:15 2021 +1000 Delete the fork service commit e63b5de Author: Michael Sproul <[email protected]> Date: Thu Jul 22 12:11:43 2021 +1000 Altair validator client and API commit 2b04ddf Merge: 59a32a3 923486f Author: pawan <[email protected]> Date: Wed Jul 28 14:45:57 2021 +0530 Merge branch 'unstable' into network-altair commit 59a32a3 Author: pawan <[email protected]> Date: Sat Jul 24 00:18:47 2021 +0530 Fix minor issues commit 0656056 Author: pawan <[email protected]> Date: Fri Jul 23 18:26:57 2021 +0530 Change libp2p dependency to sigp fork commit bed92c9 Author: realbigsean <[email protected]> Date: Tue Jul 13 12:21:56 2021 -0400 fix incorrect method name and update test to ensure off-by-one is invalid commit 6c76853 Merge: 2e35733 63923ea Author: pawan <[email protected]> Date: Wed Jul 21 23:31:30 2021 +0530 Merge discv5 upstream changes commit 2e35733 Author: pawan <[email protected]> Date: Tue Jul 20 16:14:34 2021 +0530 Maintain sync committee peers commit 9f29788 Author: pawan <[email protected]> Date: Mon Jul 19 15:42:12 2021 +0530 Fix subnet service test commit 9f9c184 Merge: 3d42f06 9a8320b Author: pawan <[email protected]> Date: Fri Jul 16 20:44:09 2021 +0530 Merge network-1.5 changes commit 3d42f06 Author: Pawan Dhananjay <[email protected]> Date: Fri Jul 16 18:56:38 2021 +0530 Network altair gossip (#2464) * small refactors * Update attestation/deposit processing * Implement `upgrade_to_altair`, genesis init test * fix slashings and implement sync committee processing * refactor common epoch processing methods * delete unused imports * Fix typo in sync_committee_updates * Update to v1.1.0-alpha.3 * Fix remaining EF tests/handlers! * Fix beacon chain tests non-altair (#2305) * Use RefMut * Reset Cargo.lock to "stable" style * Tidy block tree hash * Temporarily hardcode base fork in op pool * Update block proposal & op pool * Fix bounds check on compute_sync_committee_indices * Tidy phase0 op pool * Remove global fork schedule (#2314) * Remove global fork schedule * Fix remaining EF tests for refactor * Get top-level compiling again * Bubble YAML config changes * Use superstruct partial getters This makes our Altair consensus logic compatible with future forks. * Lots of test refactors (#2315) * Fix beacon chain tests non-altair * fix op pool tests * fix merge * get block processing tests working * Refactor state transition vector tests, half are passing * cargo fmt * PR updates * Skip ahead to epoch 256 on beacon harness tests. Fix op pool tests. * clippy updates * More test cleanups and fixes * Use `upgrade_to_altair` in state transition * More test fixes * Begin adding pools related to sync committees * Add fork upgrade test, fix EF spec test * Sync committee pool refactors, attempt to get ssz tests working * Revert BitVector change * Update lcli and local testnet scripts * Fix `SyncCommitteeContribution` bit vector length, fix `SyncCommitteeSignature` field name * update docs and add tests for sync committee operation pool * More test updates, release-tests might pass now... * Nuke old store benchmark * Fix arbitrary instances * Participation bits fixes, clippy fixes * add BitVector intersection and union tests * Disable indexing in types & state processing * Delete tree hash and state proc benchmarks * Disable aggressive lints in tests * Appease udeps * Fix the RPC tests without hacks Thanks @pawanjay176 * Fix double update in github cargo fmt * Use published superstruct * Try splitting beacon_chain tests out of CI * Fix cargo-fmt --all * Abstract over fork in beacon_chain tests Also consolidate persistence_tests into store tests to reduce the # of integration test binaries slightly. * Fix clippy * Maintain backwards compatibility with std config * Sync committee signature verification work * Add an `Arc` to the `current_sync_committee` field, refactor `sync_committee_verification` code * Clean up imports * Move `sync_committee_base_epoch` to `Epoch` * Fmt and fix some lints * Add tests for new observation pools * Some op pool work * Provide clarity about BeaconState leaf count (#2339) * Remove sync committee indices cache We could add back a cache at some point, but it would likely look completely different due to having to compute the indices "backwards" from the public keys. Dealing with all the cases involving rebuilding the cache efficiently, or what to do if it isn't built will be quite involved so I think it's better we wait to see if we need it before optimising prematurely. * - Add beacon harness methods for sync signature create - separate sync signature `SubnetId` and `SelecitonProof` types from attestations - update op pool persistence to include sync aggregates * merge with `altair` * Add tests for decoding state/block with bad slot (#2341) * Tidy SyncCommittee types * More sync committee type cleanups/fallout * merge with `altair` * Some sync verification fixes and test progress * Add hack to allow harness with altair spec * Add test for applying base block to altair chain * Use graffiti len more consistently (#2348) * Replace a test deleted during Altair update (#2351) * Apply comment fixes from Paul Co-authored-by: Paul Hauner <[email protected]> * Add fork_name methods * Add fork checks and tests * Add tests for applying altair to base * Allow providing a custom spec to test harness * Fix compile error in network * Add tests for slot, epoch processing * Fix clippy lints * Fix release test * Finish op pool sync committee logic, add a couple tests. * Avoid slicing in PartialBeaconState (#2361) * [Altair] Make anonymous var-length SSZ decoding explicit (#2362) * Add explicit function for anon var length items * Remove unused import * Update comment * Use macros for naive aggregation pool tests. * small updates * [Altair] Rewards tests (#2349) * Rewards tests (base-only so far) * Altair rewards tests, clean-ups * Bump Rust version in Dockerfile * [Altair] Remove duplicate vars, panic on `u64::pow` overflow (#2350) * [Altair] Applying Base block to Altair chain (#2353) * Add hack to allow harness with altair spec * Add test for applying base block to altair chain * Add fork_name methods * Add fork checks and tests * Add tests for applying altair to base * Allow providing a custom spec to test harness * Fix compile error in network * Add tests for slot, epoch processing * Fix clippy lints * Fix release test * Fix test compile error * Fix test with inconsistent spec * Use fork_name_at_slot in genesis * [Altair] Add check for missed EF tests (#2346) * Rewards tests (base-only so far) * Altair rewards tests, clean-ups * Bump Rust version in Dockerfile * Add file access logging * Tidy * Fix clippy lint * Fix typo * Tidy, remove commented-out test exceptions * Improve naming in python script Co-authored-by: Michael Sproul <[email protected]> * Fix beacon chain harness bug in `make_sync_contributions`. Key the `ObservedSyncAggregators` pool on slot + subcommittee index * Accidentally removed a couple constants while merging * fix a typo in a const, and a bunch of lints * Restore lost CI YAML * Address review comments * Update comment * Return an error if total bal is 0 (#2366) * Fix and test selection proof `verify` * Fix get_sync_aggregate * update op pool tests to correctly use `get_sync_aggregate` * Minor changes to justification code (#2367) * [Altair] Optimization for `get_unslashed_participating_indices` (#2369) * Lift calculation of unslashed indices * Use HashSet for unslashed participating indices * Make Operation Pool Compile Again (MOPCA) * Add chainspec and genesis validators root to eth2_libp2p * Network altair rpc (#2301) * Add v2 messages to rpc decoder * Ugly hack * Pass chainspec and genesis_root to Rpc * Add context bytes while encoding * Add a ForkContext struct * Pass ForkContext to rpc * crate compiles * Extract ForkContext into separate file; add a current_fork field * Fix encoding/decoding * Fix tests * Remove fork_schedule from rebase * Fix ForkContext * Fix tests * Remove fork_schedule again * Add altair empty and full block limits * Fix panic in snappy decoding * Fix limits * Move wrapping of RPCRequests to handler * RpcRequestContainer only used in OutboundUpgrade * Add altair blocks in rpc end to end tests * same rpc limits for V1 and V2 * V2 response decoding happens only for valid protocols * Add snappy response decoding tests * Add more snappy tests * Minor fixes * Appease clippy * to_context_bytes returns an Option * Add padding snappy message test for v2 * Minor fixes; remove accidentally added file * lint * Add chainspec and genesis validators root to eth2_libp2p * Add altair gossip types * Subscribe to required gossip topics * Unsubscribe from pre fork topics * Compiles * Start adding sync committee attnets * Add Subnet enum to include SyncCommittee subnet_ids * update_enr_bitfield takes a Subnet instead of SubnetId * Fix network service * Add MetaData V2 * Don't read context bytes for MetaData V2 * Add sync committee topics to whitelist * Fix metadata encoding; fmt * Fix next_fork methods in chainspec * Minor fixes * Fix next_fork_update in network service; add next_unsubscribe delay * Accept peers which haven't updated next_fork_version/epoch * Add sync committee topic tests * Simplify required_gossip_digests; improve logging around fork boundary * Fix tests * Minor refactorings * Add sync contribution aggregation tests to the op pool and fix an op pool bug * Only advance state as necessary in sync committee verification tests. Fix compile error. * Delete `verify_sync_contributions.rs` and fix a comment. * Remove `Arc` from `current_sync_committee` * Add `max_capacity` to `ObservedAggregates` constructor * Update consensus code to v1.1.0-alpha.6 * Make the `SyncAggregateId` a fixed-length struct * Add metrics and update docs * Address review comments * Sync committee subnets * Ripple spec updates * Revert "Remove `Arc` from `current_sync_committee`" This reverts commit bf6324e. * fix some pool tests * Updates to some docs, general cleanup * Fix default capacity for sync committee observed caches * Fix default capacity for sync committee observed caches * Fix default capacity for sync committee observed caches * Fix more docs and metrics * Remove `is_valid` methods from `SignedAggregateAndProof` and `SignedContributionAndProof`. The logic is duplicated in the signature set verification methods. * remove outdated `FIXME` * Fix docs * Fix docs * Implement preset <> config distinction * Update confusing name `SyncCommitteeSubnetSize` -> `SyncSubcommitteeSize` * Updates for latest spec merge * update client for `lighthouse/spec` endpoint * Add back accidentally deleted import * Return ConfigAndPreset from VC API * Don't panic if there aren't any aggregators Rustfmt also returned with a vengeance to rearrange this block. * Check state root in transition tests * Fix clippy * Fix Pyrmont & Prater configs. Delete Toledo. * Add compatibility shim for /config/spec * Fix fork_digest in Status message * Revert "update client for `lighthouse/spec` endpoint" This reverts commit 7a9b2ac. # Conflicts: # validator_client/src/http_api/tests.rs * - Update `SYNC_COMMITTEE_SUBNET_COUNT` to alpha.6 - rename the `sync_aggregator_selection_data.rs` file - include sync committee ssz static tests in the `check_all_file_accessed.py` checks - Update `ObservedAggregates` consts so they are specific to the type of object being cached * - Update `NaiveAggregationPool`'s default capacity so it is specific to the type of object being inserted * fix `observed_aggregates` tests * Add chainspec and genesis validators root to eth2_libp2p * Network altair rpc (#2301) * Add v2 messages to rpc decoder * Ugly hack * Pass chainspec and genesis_root to Rpc * Add context bytes while encoding * Add a ForkContext struct * Pass ForkContext to rpc * crate compiles * Extract ForkContext into separate file; add a current_fork field * Fix encoding/decoding * Fix tests * Remove fork_schedule from rebase * Fix ForkContext * Fix tests * Remove fork_schedule again * Add altair empty and full block limits * Fix panic in snappy decoding * Fix limits * Move wrapping of RPCRequests to handler * RpcRequestContainer only used in OutboundUpgrade * Add altair blocks in rpc end to end tests * same rpc limits for V1 and V2 * V2 response decoding happens only for valid protocols * Add snappy response decoding tests * Add more snappy tests * Minor fixes * Appease clippy * to_context_bytes returns an Option * Add padding snappy message test for v2 * Minor fixes; remove accidentally added file * lint * Use SyncSubnetId for sync committee subnets * Add worker tasks for sync committee gossip messages * Use correct type level constants * Decode altair gossip blocks * Add a SyncCommitteeSubscribe network event * PR updates * fix op pool tests * Minor cleanups * Calculate `SyncAggregate` on demand * Address review comments * Op pool migration + superstruct * Fix op pool test * Update to spec v1.1.0-alpha.7 * Fix sync committee subscriptions * Add sync committee subscription test * Add chainspec and genesis validators root to eth2_libp2p * Network altair rpc (#2301) * Add v2 messages to rpc decoder * Ugly hack * Pass chainspec and genesis_root to Rpc * Add context bytes while encoding * Add a ForkContext struct * Pass ForkContext to rpc * crate compiles * Extract ForkContext into separate file; add a current_fork field * Fix encoding/decoding * Fix tests * Remove fork_schedule from rebase * Fix ForkContext * Fix tests * Remove fork_schedule again * Add altair empty and full block limits * Fix panic in snappy decoding * Fix limits * Move wrapping of RPCRequests to handler * RpcRequestContainer only used in OutboundUpgrade * Add altair blocks in rpc end to end tests * same rpc limits for V1 and V2 * V2 response decoding happens only for valid protocols * Add snappy response decoding tests * Add more snappy tests * Minor fixes * Appease clippy * to_context_bytes returns an Option * Add padding snappy message test for v2 * Minor fixes; remove accidentally added file * lint * - Fix sync committee verification on slot prior to sync committee period boundary - Add `Arc` to `next_sync_committee` - Other PR updates * Refactor how we retrieve `PublicKey` for sync signature verification * Key `observed_contributors` on `SlotSubcommitteeIndex` * Update mainnet preset YAML * Gossipsub message_id_fn changes * Add tests for sync committee verification at slot before sync committee period boundary * fix op pool tests * - Make `SyncSubnetId` a required argument for `verify_sync_signature_for_gossip` - Drop locks as soon as possible during metrics scrape * fix sync committee verification test macro * Fix beacon_chain test * rust 1.53.0 updates * add recursion limit to `simulator` crate * cargo fmt * `SyncCommitteSignature` -> `SyncCommitteeMessage` * Updates related to merge with `unstable` * More updates related to `SyncCommitteMessage` refactor * Couple more `unstable` merge fixes * Fix `SyncAggregate` field name * Add chainspec and genesis validators root to eth2_libp2p * Network altair rpc (#2301) * Add v2 messages to rpc decoder * Ugly hack * Pass chainspec and genesis_root to Rpc * Add context bytes while encoding * Add a ForkContext struct * Pass ForkContext to rpc * crate compiles * Extract ForkContext into separate file; add a current_fork field * Fix encoding/decoding * Fix tests * Remove fork_schedule from rebase * Fix ForkContext * Fix tests * Remove fork_schedule again * Add altair empty and full block limits * Fix panic in snappy decoding * Fix limits * Move wrapping of RPCRequests to handler * RpcRequestContainer only used in OutboundUpgrade * Add altair blocks in rpc end to end tests * same rpc limits for V1 and V2 * V2 response decoding happens only for valid protocols * Add snappy response decoding tests * Add more snappy tests * Minor fixes * Appease clippy * to_context_bytes returns an Option * Add padding snappy message test for v2 * Minor fixes; remove accidentally added file * lint * Change libp2p dependency for message_id_fn * appease clippy * Add sync committee related metrics * Expand and refactor fork functions * Simplify compute_subnets_for_sync_committee * Fix subnet service test * Add v2 protocols for OutboundRequests missed in the merge * Handle sync committee gossip errors * Remove delay for updating fork * fix the expected error in the sync contribution verification test at the sync period boundary * Updates related to merge with `unstable` * Couple more `unstable` merge fixes * Split rpc encode/decode into smaller functions * Import some relevant funcs from the VC Based on "Expand and refactor fork functions" * Fix issues with skip slots at fork boundaries * Fix clippy again * alpha.8 updates * Pop sync committee tasks from beacon processor * Fix some gossip scoring severity issue * Fix bug in peer_manager; Improve logging around subnet discoveries Co-authored-by: realbigsean <[email protected]> Co-authored-by: Michael Sproul <[email protected]> Co-authored-by: realbigsean <[email protected]> Co-authored-by: Paul Hauner <[email protected]> commit f37f267 Author: Pawan Dhananjay <[email protected]> Date: Tue May 11 10:30:43 2021 +0530 Network altair rpc (#2301) * Add v2 messages to rpc decoder * Ugly hack * Pass chainspec and genesis_root to Rpc * Add context bytes while encoding * Add a ForkContext struct * Pass ForkContext to rpc * crate compiles * Extract ForkContext into separate file; add a current_fork field * Fix encoding/decoding * Fix tests * Remove fork_schedule from rebase * Fix ForkContext * Fix tests * Remove fork_schedule again * Add altair empty and full block limits * Fix panic in snappy decoding * Fix limits * Move wrapping of RPCRequests to handler * RpcRequestContainer only used in OutboundUpgrade * Add altair blocks in rpc end to end tests * same rpc limits for V1 and V2 * V2 response decoding happens only for valid protocols * Add snappy response decoding tests * Add more snappy tests * Minor fixes * Appease clippy * to_context_bytes returns an Option * Add padding snappy message test for v2 * Minor fixes; remove accidentally added file * lint commit b477bdd Author: pawan <[email protected]> Date: Mon Apr 5 19:30:48 2021 +0530 Add chainspec and genesis validators root to eth2_libp2p

That was *not* doing what I thought it was doing!

paulhauner

Looks great, I could hardly fault it! I suspect the changes I suggested in the comments are just copy-paste from my previous comments, which were already wrong 😅

I quite like the aggregate filling-in mechanism.

beacon_node/http_api/src/sync_committees.rs

paulhauner · 2021-08-05T08:22:24Z

validator_client/src/sync_committee_service.rs

+        );
+
+        let aggregators = slot_duties.aggregators;
+        self.inner.context.executor.spawn(


In the attestation service we wait until the unaggregate is published before we produce the aggregate. This is primarily because the gossip validation spec for aggregates states:

The aggregator's validator index is within the committee

This is not the case for sync aggregates, so there's no hard requirement here to block aggs on the unaggs (aka sync sigs). It does raise the question of motives, though. A selfish validator would probably want to ensure they wait until the BN has their sync sig before creating a sync agg. However, a nice-guy validator might want to ensure the network gets their aggregates, even if their own was a bit late.

This is an edge case and you've taken then nice-guy route here (I wouldn't expect any less from ya *slaps back*). I'm happy to leave this as-is, I just wanted to share my reasoning. Also, I haven't found much yet and this shows I'm actually thinking about things 😅

Oh that's interesting, and quite subtle. I don't think I explicitly considered this selfish/altruistic tradeoff – I think it might have just come about as a side-effect of restructuring the attestation logic to support sync messages. The main change required was moving from a subnet-centric view of everything to only caring about subnets for the aggregated messages (because the unaggregated messages need to be broadcast once, and then published to multiple subnets by the BN). That change split the two tasks into different spawn invocations, but it wouldn't be impossible to merge them together again.

The VC should be able to publish the unaggregated messages in seconds 4-8 of the slot, so I'm tempted to leave this as-is for now, and reassess if we have issues?

I agree with leaving this as-is for now. It's simpler, a rare edge-case (AFAICT), beneficial to the network and unlikely to be that bad for a validator.

validator_client/src/duties_service/sync.rs

paulhauner

🚀

michaelsproul · 2021-08-06T00:47:16Z

Yay! Thanks!

bors r+

## Proposed Changes * Implement the validator client and HTTP API changes necessary to support Altair Co-authored-by: realbigsean <[email protected]> Co-authored-by: Michael Sproul <[email protected]>

bors · 2021-08-06T02:13:11Z

Pull request successfully merged into unstable.

Build succeeded:

michaelsproul · 2021-08-06T02:17:00Z

Holy shit! Finally!

michaelsproul added the work-in-progress PR is a work-in-progress label Jun 10, 2021

michaelsproul changed the title ~~Altair vc~~ Altair validator client updates Jun 10, 2021

paulhauner mentioned this pull request Jul 7, 2021

Altair-devnet-0 #2434

Closed

michaelsproul commented Jul 9, 2021

View reviewed changes

validator_client/src/duties_service/sync.rs Outdated Show resolved Hide resolved

michaelsproul changed the base branch from altair to network-altair July 15, 2021 01:22

michaelsproul marked this pull request as ready for review July 15, 2021 01:28

michaelsproul changed the title ~~Altair validator client updates~~ Altair validator client and HTTP API Jul 15, 2021

pawanjay176 added 2 commits July 16, 2021 17:58

Add chainspec and genesis validators root to eth2_libp2p

b477bdd

pawanjay176 force-pushed the network-altair branch from 87945c2 to f37f267 Compare July 16, 2021 13:21

pawanjay176 and others added 3 commits July 16, 2021 18:56

Merge network-1.5 changes

9f9c184

Fix subnet service test

9f29788

pawanjay176 force-pushed the network-altair branch from f54a086 to 9f29788 Compare July 19, 2021 10:12

pawanjay176 added 2 commits July 20, 2021 18:41

Maintain sync committee peers

2e35733

Merge discv5 upstream changes

6c76853

michaelsproul added the v1.5.0 For inclusion in v1.5.0 release label Jul 22, 2021

michaelsproul force-pushed the altair-vc branch from 0190506 to 0f5593c Compare July 22, 2021 03:13

realbigsean and others added 3 commits July 22, 2021 12:17

fix incorrect method name and update test to ensure off-by-one is inv…

bed92c9

…alid

Change libp2p dependency to sigp fork

0656056

Fix minor issues

59a32a3

michaelsproul force-pushed the altair-vc branch from 0f5593c to 9ed0063 Compare July 26, 2021 08:29

michaelsproul mentioned this pull request Jul 26, 2021

Altair devnets #2450

Closed

pawanjay176 and others added 3 commits July 28, 2021 14:46

Merge branch 'unstable' into network-altair

2b04ddf

Altair validator client and API

e63b5de

Delete the fork service

d0e2153

michaelsproul and others added 5 commits August 3, 2021 17:13

Merge remote-tracking branch 'origin/network-altair' into altair-vc

86b68a4

Fix clippy

3266fcd

Address more review comments

6a6d175

Fix beacon chain tests

8e7466a

Merge remote-tracking branch 'origin/network-altair' into altair-vc

410d751

michaelsproul added ready-for-review The code is ready for review and removed waiting-on-author The reviewer has suggested changes and awaits thier implementation. labels Aug 3, 2021

Fix unset SPEC_PRESET issue in doppel test

df3fab1

michaelsproul changed the base branch from network-altair to unstable August 4, 2021 04:53

michaelsproul added 3 commits August 4, 2021 15:17

Merge remote-tracking branch 'origin/unstable' into altair-vc

64555fd

Clean up discv5 dev dependencies

23a7839

Fix bash set invocation

077eab6

That was *not* doing what I thought it was doing!

michaelsproul force-pushed the altair-vc branch from a7d341b to 077eab6 Compare August 5, 2021 01:55

paulhauner requested changes Aug 5, 2021

View reviewed changes

Address some review comments

2dd396c

paulhauner approved these changes Aug 6, 2021

View reviewed changes

paulhauner added ready-for-merge This PR is ready to merge. and removed ready-for-review The code is ready for review labels Aug 6, 2021

bors bot changed the title ~~Altair validator client and HTTP API~~ [Merged by Bors] - Altair validator client and HTTP API Aug 6, 2021

bors bot closed this Aug 6, 2021

michaelsproul deleted the altair-vc branch August 6, 2021 08:24

paulhauner mentioned this pull request Oct 1, 2021

Disable autotests for beacon_chain crate #2658

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Merged by Bors] - Altair validator client and HTTP API #2404

[Merged by Bors] - Altair validator client and HTTP API #2404

michaelsproul commented Jun 10, 2021

michaelsproul commented Jun 10, 2021 •

edited

Loading

michaelsproul commented Jun 10, 2021

michaelsproul commented Jul 22, 2021

michaelsproul commented Jul 26, 2021

michaelsproul commented Aug 3, 2021

michaelsproul commented Aug 4, 2021

paulhauner left a comment

paulhauner Aug 5, 2021

michaelsproul Aug 6, 2021

paulhauner Aug 6, 2021

paulhauner left a comment

michaelsproul commented Aug 6, 2021

bors bot commented Aug 6, 2021

michaelsproul commented Aug 6, 2021

[Merged by Bors] - Altair validator client and HTTP API #2404

[Merged by Bors] - Altair validator client and HTTP API #2404

Conversation

michaelsproul commented Jun 10, 2021

Proposed Changes

michaelsproul commented Jun 10, 2021 • edited Loading

michaelsproul commented Jun 10, 2021

michaelsproul commented Jul 22, 2021

michaelsproul commented Jul 26, 2021

michaelsproul commented Aug 3, 2021

michaelsproul commented Aug 4, 2021

paulhauner left a comment

Choose a reason for hiding this comment

paulhauner Aug 5, 2021

Choose a reason for hiding this comment

michaelsproul Aug 6, 2021

Choose a reason for hiding this comment

paulhauner Aug 6, 2021

Choose a reason for hiding this comment

paulhauner left a comment

Choose a reason for hiding this comment

michaelsproul commented Aug 6, 2021

bors bot commented Aug 6, 2021

michaelsproul commented Aug 6, 2021

michaelsproul commented Jun 10, 2021 •

edited

Loading