Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modular Reqresp Package #4775

Merged
merged 23 commits into from
Nov 20, 2022
Merged

Modular Reqresp Package #4775

merged 23 commits into from
Nov 20, 2022

Conversation

nazarhussain
Copy link
Contributor

Motivation

Make the reqresp modular so it can be consumed among light-client and beacon-node easily.

Description

Extract the reqresp out of beacon node and create its own package. Made the messages and handlers generic.

Steps to test or reproduce

Run unit tests.

@nazarhussain nazarhussain self-assigned this Nov 17, 2022
@@ -0,0 +1,15 @@
/*
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this file can be deleted

LightClientBootstrap = "light_client_bootstrap",
LightClientUpdatesByRange = "light_client_updates_by_range",
LightClientFinalityUpdate = "light_client_finality_update",
LightClientOptimisticUpdate = "light_client_optimistic_update",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reqResp package is agnostic of full node specifics. All of this should be in the beacon-node package only

@dapplion dapplion force-pushed the dapplion/reqresp-modular branch from 8cca03b to 38b9225 Compare November 18, 2022 14:44
@github-actions
Copy link
Contributor

github-actions bot commented Nov 18, 2022

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: 707dcdd Previous: f74462e Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 2.9186 ms/op 2.2135 ms/op 1.32
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 114.24 us/op 83.163 us/op 1.37
BLS verify - blst-native 2.3664 ms/op 2.2073 ms/op 1.07
BLS verifyMultipleSignatures 3 - blst-native 4.9141 ms/op 4.5370 ms/op 1.08
BLS verifyMultipleSignatures 8 - blst-native 10.314 ms/op 9.7452 ms/op 1.06
BLS verifyMultipleSignatures 32 - blst-native 37.227 ms/op 34.876 ms/op 1.07
BLS aggregatePubkeys 32 - blst-native 50.943 us/op 49.570 us/op 1.03
BLS aggregatePubkeys 128 - blst-native 194.71 us/op 181.05 us/op 1.08
getAttestationsForBlock 126.43 ms/op 105.02 ms/op 1.20
isKnown best case - 1 super set check 524.00 ns/op 508.00 ns/op 1.03
isKnown normal case - 2 super set checks 515.00 ns/op 497.00 ns/op 1.04
isKnown worse case - 16 super set checks 529.00 ns/op 497.00 ns/op 1.06
CheckpointStateCache - add get delete 11.665 us/op 10.134 us/op 1.15
validate gossip signedAggregateAndProof - struct 5.5819 ms/op 5.0212 ms/op 1.11
validate gossip attestation - struct 2.5967 ms/op 2.3755 ms/op 1.09
pickEth1Vote - no votes 2.6935 ms/op 2.4624 ms/op 1.09
pickEth1Vote - max votes 27.053 ms/op 22.356 ms/op 1.21
pickEth1Vote - Eth1Data hashTreeRoot value x2048 14.813 ms/op 12.508 ms/op 1.18
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 26.325 ms/op 23.943 ms/op 1.10
pickEth1Vote - Eth1Data fastSerialize value x2048 1.8981 ms/op 1.9171 ms/op 0.99
pickEth1Vote - Eth1Data fastSerialize tree x2048 18.300 ms/op 15.373 ms/op 1.19
bytes32 toHexString 1.4090 us/op 1.1540 us/op 1.22
bytes32 Buffer.toString(hex) 904.00 ns/op 760.00 ns/op 1.19
bytes32 Buffer.toString(hex) from Uint8Array 1.2130 us/op 1.0210 us/op 1.19
bytes32 Buffer.toString(hex) + 0x 895.00 ns/op 786.00 ns/op 1.14
Object access 1 prop 0.47700 ns/op 0.40900 ns/op 1.17
Map access 1 prop 0.37700 ns/op 0.34200 ns/op 1.10
Object get x1000 17.519 ns/op 20.419 ns/op 0.86
Map get x1000 1.0230 ns/op 1.1440 ns/op 0.89
Object set x1000 122.57 ns/op 127.52 ns/op 0.96
Map set x1000 91.535 ns/op 82.194 ns/op 1.11
Return object 10000 times 0.45440 ns/op 0.43980 ns/op 1.03
Throw Error 10000 times 7.1339 us/op 6.8949 us/op 1.03
fastMsgIdFn sha256 / 200 bytes 5.4800 us/op 4.7980 us/op 1.14
fastMsgIdFn h32 xxhash / 200 bytes 647.00 ns/op 612.00 ns/op 1.06
fastMsgIdFn h64 xxhash / 200 bytes 870.00 ns/op 769.00 ns/op 1.13
fastMsgIdFn sha256 / 1000 bytes 16.850 us/op 15.095 us/op 1.12
fastMsgIdFn h32 xxhash / 1000 bytes 852.00 ns/op 769.00 ns/op 1.11
fastMsgIdFn h64 xxhash / 1000 bytes 934.00 ns/op 900.00 ns/op 1.04
fastMsgIdFn sha256 / 10000 bytes 147.10 us/op 126.76 us/op 1.16
fastMsgIdFn h32 xxhash / 10000 bytes 2.8750 us/op 2.7450 us/op 1.05
fastMsgIdFn h64 xxhash / 10000 bytes 2.2500 us/op 2.1000 us/op 1.07
enrSubnets - fastDeserialize 64 bits 3.1650 us/op 2.9060 us/op 1.09
enrSubnets - ssz BitVector 64 bits 976.00 ns/op 822.00 ns/op 1.19
enrSubnets - fastDeserialize 4 bits 490.00 ns/op 425.00 ns/op 1.15
enrSubnets - ssz BitVector 4 bits 950.00 ns/op 872.00 ns/op 1.09
prioritizePeers score -10:0 att 32-0.1 sync 2-0 118.35 us/op 105.66 us/op 1.12
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 159.98 us/op 145.36 us/op 1.10
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 298.63 us/op 242.88 us/op 1.23
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 651.51 us/op 516.50 us/op 1.26
prioritizePeers score 0:0 att 64-1 sync 4-1 592.20 us/op 504.90 us/op 1.17
RateTracker 1000000 limit, 1 obj count per request 220.29 ns/op 224.66 ns/op 0.98
RateTracker 1000000 limit, 2 obj count per request 166.09 ns/op 172.85 ns/op 0.96
RateTracker 1000000 limit, 4 obj count per request 138.53 ns/op 143.96 ns/op 0.96
RateTracker 1000000 limit, 8 obj count per request 122.54 ns/op 128.27 ns/op 0.96
RateTracker with prune 6.1020 us/op 4.7630 us/op 1.28
array of 16000 items push then shift 5.7252 us/op 3.5780 us/op 1.60
LinkedList of 16000 items push then shift 21.980 ns/op 19.216 ns/op 1.14
array of 16000 items push then pop 274.42 ns/op 261.45 ns/op 1.05
LinkedList of 16000 items push then pop 20.353 ns/op 19.191 ns/op 1.06
array of 24000 items push then shift 8.6258 us/op 5.3116 us/op 1.62
LinkedList of 24000 items push then shift 21.568 ns/op 22.403 ns/op 0.96
array of 24000 items push then pop 268.55 ns/op 236.86 ns/op 1.13
LinkedList of 24000 items push then pop 20.515 ns/op 20.110 ns/op 1.02
intersect bitArray bitLen 8 13.120 ns/op 13.418 ns/op 0.98
intersect array and set length 8 228.60 ns/op 186.47 ns/op 1.23
intersect bitArray bitLen 128 78.572 ns/op 71.531 ns/op 1.10
intersect array and set length 128 2.6097 us/op 2.6161 us/op 1.00
Buffer.concat 32 items 2.4780 ns/op 2.1890 ns/op 1.13
pass gossip attestations to forkchoice per slot 4.8438 ms/op 4.9457 ms/op 0.98
computeDeltas 6.0899 ms/op 7.0619 ms/op 0.86
computeProposerBoostScoreFromBalances 939.46 us/op 1.0677 ms/op 0.88
altair processAttestation - 250000 vs - 7PWei normalcase 5.8773 ms/op 4.2034 ms/op 1.40
altair processAttestation - 250000 vs - 7PWei worstcase 8.4476 ms/op 6.6795 ms/op 1.26
altair processAttestation - setStatus - 1/6 committees join 278.07 us/op 246.22 us/op 1.13
altair processAttestation - setStatus - 1/3 committees join 500.40 us/op 471.10 us/op 1.06
altair processAttestation - setStatus - 1/2 committees join 714.38 us/op 678.98 us/op 1.05
altair processAttestation - setStatus - 2/3 committees join 951.79 us/op 863.19 us/op 1.10
altair processAttestation - setStatus - 4/5 committees join 1.3769 ms/op 1.1697 ms/op 1.18
altair processAttestation - setStatus - 100% committees join 1.6470 ms/op 1.4223 ms/op 1.16
altair processBlock - 250000 vs - 7PWei normalcase 33.718 ms/op 30.740 ms/op 1.10
altair processBlock - 250000 vs - 7PWei normalcase hashState 43.134 ms/op 41.368 ms/op 1.04
altair processBlock - 250000 vs - 7PWei worstcase 115.21 ms/op 103.37 ms/op 1.11
altair processBlock - 250000 vs - 7PWei worstcase hashState 122.66 ms/op 108.54 ms/op 1.13
phase0 processBlock - 250000 vs - 7PWei normalcase 4.8468 ms/op 4.1445 ms/op 1.17
phase0 processBlock - 250000 vs - 7PWei worstcase 59.883 ms/op 55.728 ms/op 1.07
altair processEth1Data - 250000 vs - 7PWei normalcase 1.1600 ms/op 980.51 us/op 1.18
Tree 40 250000 create 1.1452 s/op 918.35 ms/op 1.25
Tree 40 250000 get(125000) 343.65 ns/op 348.65 ns/op 0.99
Tree 40 250000 set(125000) 3.3893 us/op 2.7763 us/op 1.22
Tree 40 250000 toArray() 38.460 ms/op 36.421 ms/op 1.06
Tree 40 250000 iterate all - toArray() + loop 37.936 ms/op 36.417 ms/op 1.04
Tree 40 250000 iterate all - get(i) 141.80 ms/op 128.49 ms/op 1.10
MutableVector 250000 create 18.185 ms/op 17.667 ms/op 1.03
MutableVector 250000 get(125000) 15.119 ns/op 15.410 ns/op 0.98
MutableVector 250000 set(125000) 993.74 ns/op 733.78 ns/op 1.35
MutableVector 250000 toArray() 8.1148 ms/op 8.7205 ms/op 0.93
MutableVector 250000 iterate all - toArray() + loop 8.2630 ms/op 8.5947 ms/op 0.96
MutableVector 250000 iterate all - get(i) 3.5746 ms/op 3.9225 ms/op 0.91
Array 250000 create 9.5270 ms/op 7.8033 ms/op 1.22
Array 250000 clone - spread 3.8742 ms/op 4.3960 ms/op 0.88
Array 250000 get(125000) 1.5840 ns/op 1.8360 ns/op 0.86
Array 250000 set(125000) 1.5370 ns/op 1.8920 ns/op 0.81
Array 250000 iterate all - loop 151.47 us/op 205.88 us/op 0.74
effectiveBalanceIncrements clone Uint8Array 300000 246.94 us/op 106.92 us/op 2.31
effectiveBalanceIncrements clone MutableVector 300000 804.00 ns/op 1.5110 us/op 0.53
effectiveBalanceIncrements rw all Uint8Array 300000 308.23 us/op 304.51 us/op 1.01
effectiveBalanceIncrements rw all MutableVector 300000 241.87 ms/op 236.23 ms/op 1.02
phase0 afterProcessEpoch - 250000 vs - 7PWei 228.62 ms/op 234.29 ms/op 0.98
phase0 beforeProcessEpoch - 250000 vs - 7PWei 84.932 ms/op 78.695 ms/op 1.08
altair processEpoch - mainnet_e81889 683.57 ms/op 676.61 ms/op 1.01
mainnet_e81889 - altair beforeProcessEpoch 167.19 ms/op 175.98 ms/op 0.95
mainnet_e81889 - altair processJustificationAndFinalization 65.678 us/op 40.336 us/op 1.63
mainnet_e81889 - altair processInactivityUpdates 12.822 ms/op 12.351 ms/op 1.04
mainnet_e81889 - altair processRewardsAndPenalties 110.49 ms/op 103.79 ms/op 1.06
mainnet_e81889 - altair processRegistryUpdates 14.197 us/op 8.3300 us/op 1.70
mainnet_e81889 - altair processSlashings 3.9140 us/op 1.8580 us/op 2.11
mainnet_e81889 - altair processEth1DataReset 3.7830 us/op 2.3550 us/op 1.61
mainnet_e81889 - altair processEffectiveBalanceUpdates 2.4606 ms/op 3.1250 ms/op 0.79
mainnet_e81889 - altair processSlashingsReset 24.661 us/op 13.497 us/op 1.83
mainnet_e81889 - altair processRandaoMixesReset 21.489 us/op 12.481 us/op 1.72
mainnet_e81889 - altair processHistoricalRootsUpdate 3.5260 us/op 2.0030 us/op 1.76
mainnet_e81889 - altair processParticipationFlagUpdates 11.529 us/op 8.0530 us/op 1.43
mainnet_e81889 - altair processSyncCommitteeUpdates 2.7110 us/op 1.7740 us/op 1.53
mainnet_e81889 - altair afterProcessEpoch 224.94 ms/op 228.00 ms/op 0.99
phase0 processEpoch - mainnet_e58758 630.19 ms/op 610.51 ms/op 1.03
mainnet_e58758 - phase0 beforeProcessEpoch 288.73 ms/op 264.81 ms/op 1.09
mainnet_e58758 - phase0 processJustificationAndFinalization 60.555 us/op 34.165 us/op 1.77
mainnet_e58758 - phase0 processRewardsAndPenalties 163.39 ms/op 148.44 ms/op 1.10
mainnet_e58758 - phase0 processRegistryUpdates 32.093 us/op 19.914 us/op 1.61
mainnet_e58758 - phase0 processSlashings 3.1660 us/op 1.8800 us/op 1.68
mainnet_e58758 - phase0 processEth1DataReset 3.4750 us/op 1.9620 us/op 1.77
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 2.6755 ms/op 2.8269 ms/op 0.95
mainnet_e58758 - phase0 processSlashingsReset 16.153 us/op 8.6730 us/op 1.86
mainnet_e58758 - phase0 processRandaoMixesReset 23.631 us/op 14.475 us/op 1.63
mainnet_e58758 - phase0 processHistoricalRootsUpdate 3.6320 us/op 2.5510 us/op 1.42
mainnet_e58758 - phase0 processParticipationRecordUpdates 23.047 us/op 12.467 us/op 1.85
mainnet_e58758 - phase0 afterProcessEpoch 188.18 ms/op 190.01 ms/op 0.99
phase0 processEffectiveBalanceUpdates - 250000 normalcase 2.8241 ms/op 3.1003 ms/op 0.91
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 3.2937 ms/op 4.1763 ms/op 0.79
altair processInactivityUpdates - 250000 normalcase 52.877 ms/op 46.340 ms/op 1.14
altair processInactivityUpdates - 250000 worstcase 66.054 ms/op 58.000 ms/op 1.14
phase0 processRegistryUpdates - 250000 normalcase 28.394 us/op 17.994 us/op 1.58
phase0 processRegistryUpdates - 250000 badcase_full_deposits 624.98 us/op 596.47 us/op 1.05
phase0 processRegistryUpdates - 250000 worstcase 0.5 266.89 ms/op 242.56 ms/op 1.10
altair processRewardsAndPenalties - 250000 normalcase 159.19 ms/op 144.67 ms/op 1.10
altair processRewardsAndPenalties - 250000 worstcase 101.36 ms/op 98.235 ms/op 1.03
phase0 getAttestationDeltas - 250000 normalcase 16.994 ms/op 15.439 ms/op 1.10
phase0 getAttestationDeltas - 250000 worstcase 16.643 ms/op 15.378 ms/op 1.08
phase0 processSlashings - 250000 worstcase 7.4158 ms/op 6.3625 ms/op 1.17
altair processSyncCommitteeUpdates - 250000 371.94 ms/op 324.85 ms/op 1.14
BeaconState.hashTreeRoot - No change 655.00 ns/op 584.00 ns/op 1.12
BeaconState.hashTreeRoot - 1 full validator 89.727 us/op 73.329 us/op 1.22
BeaconState.hashTreeRoot - 32 full validator 855.99 us/op 937.52 us/op 0.91
BeaconState.hashTreeRoot - 512 full validator 8.0641 ms/op 7.6245 ms/op 1.06
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 120.14 us/op 95.943 us/op 1.25
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 1.5788 ms/op 1.3951 ms/op 1.13
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 21.114 ms/op 18.297 ms/op 1.15
BeaconState.hashTreeRoot - 1 balances 80.406 us/op 73.570 us/op 1.09
BeaconState.hashTreeRoot - 32 balances 777.33 us/op 700.71 us/op 1.11
BeaconState.hashTreeRoot - 512 balances 7.2947 ms/op 6.7207 ms/op 1.09
BeaconState.hashTreeRoot - 250000 balances 122.19 ms/op 110.76 ms/op 1.10
aggregationBits - 2048 els - zipIndexesInBitList 44.283 us/op 38.978 us/op 1.14
regular array get 100000 times 59.635 us/op 79.439 us/op 0.75
wrappedArray get 100000 times 58.982 us/op 79.088 us/op 0.75
arrayWithProxy get 100000 times 34.794 ms/op 34.407 ms/op 1.01
ssz.Root.equals 641.00 ns/op 578.00 ns/op 1.11
byteArrayEquals 654.00 ns/op 594.00 ns/op 1.10
shuffle list - 16384 els 12.829 ms/op 12.598 ms/op 1.02
shuffle list - 250000 els 184.80 ms/op 188.20 ms/op 0.98
processSlot - 1 slots 21.109 us/op 13.745 us/op 1.54
processSlot - 32 slots 2.3853 ms/op 1.9910 ms/op 1.20
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 546.89 us/op 447.05 us/op 1.22
getCommitteeAssignments - req 1 vs - 250000 vc 5.6260 ms/op 6.4763 ms/op 0.87
getCommitteeAssignments - req 100 vs - 250000 vc 8.1433 ms/op 8.8883 ms/op 0.92
getCommitteeAssignments - req 1000 vs - 250000 vc 8.9082 ms/op 9.2415 ms/op 0.96
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 11.150 ns/op 11.820 ns/op 0.94
state getBlockRootAtSlot - 250000 vs - 7PWei 1.4952 us/op 1.3108 us/op 1.14
computeProposers - vc 250000 22.605 ms/op 18.850 ms/op 1.20
computeEpochShuffling - vc 250000 187.59 ms/op 192.64 ms/op 0.97
getNextSyncCommittee - vc 250000 368.92 ms/op 323.51 ms/op 1.14

by benchmarkbot/action

@@ -1,10 +0,0 @@
import {expect} from "chai";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All the unit tests are deleted?

Copy link
Contributor Author

@nazarhussain nazarhussain Nov 20, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That we discussed to add and fix the unit tests in a separate PR to not block this one.

@nazarhussain nazarhussain marked this pull request as ready for review November 20, 2022 02:14
@nazarhussain nazarhussain requested a review from a team as a code owner November 20, 2022 02:14
Copy link
Contributor

@dapplion dapplion left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! One minor comment for latter on test fixing

/**
* https://github.com/ethereum/consensus-specs/blob/v1.2.0/specs/phase0/p2p-interface.md#protocol-identification
*/
export function parseProtocolID(protocolId: string): Protocol {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function was not in source because it's only used in tests. I try to not keep any code in source that's only for testing

@dapplion dapplion merged commit 8a211da into unstable Nov 20, 2022
@dapplion dapplion deleted the dapplion/reqresp-modular branch November 20, 2022 09:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants