Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Don't track unstable LinkedList benchmark #3809

Merged
merged 1 commit into from
Mar 1, 2022
Merged

Don't track unstable LinkedList benchmark #3809

merged 1 commit into from
Mar 1, 2022

Conversation

dapplion
Copy link
Contributor

Motivation

We got the results of the data-structure performance, which is already implemented. Disable since it's failing CI builds

Description

Don't track unstable LinkedList benchmark

@codecov
Copy link

codecov bot commented Feb 28, 2022

Codecov Report

Merging #3809 (7774250) into master (62b2092) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #3809   +/-   ##
=======================================
  Coverage   36.82%   36.82%           
=======================================
  Files         324      324           
  Lines        8870     8870           
  Branches     1385     1385           
=======================================
  Hits         3266     3266           
  Misses       5461     5461           
  Partials      143      143

@github-actions
Copy link
Contributor

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: 8306bb4 Previous: 62b2092 Ratio
BeaconState.hashTreeRoot - No change 550.00 ns/op 658.00 ns/op 0.84
BeaconState.hashTreeRoot - 1 full validator 83.641 us/op 91.352 us/op 0.92
BeaconState.hashTreeRoot - 32 full validator 1.1780 ms/op 1.2924 ms/op 0.91
BeaconState.hashTreeRoot - 512 full validator 15.798 ms/op 17.166 ms/op 0.92
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 90.060 us/op 89.788 us/op 1.00
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 1.3700 ms/op 1.6283 ms/op 0.84
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 18.280 ms/op 18.589 ms/op 0.98
BeaconState.hashTreeRoot - 1 balances 57.358 us/op 66.618 us/op 0.86
BeaconState.hashTreeRoot - 32 balances 489.19 us/op 583.13 us/op 0.84
BeaconState.hashTreeRoot - 512 balances 4.5401 ms/op 5.5362 ms/op 0.82
BeaconState.hashTreeRoot - 250000 balances 91.929 ms/op 102.05 ms/op 0.90
processSlot - 1 slots 39.192 us/op 42.988 us/op 0.91
processSlot - 32 slots 2.0414 ms/op 2.2535 ms/op 0.91
getCommitteeAssignments - req 1 vs - 250000 vc 5.2604 ms/op 5.2958 ms/op 0.99
getCommitteeAssignments - req 100 vs - 250000 vc 7.3651 ms/op 7.9403 ms/op 0.93
getCommitteeAssignments - req 1000 vs - 250000 vc 7.8687 ms/op 8.3777 ms/op 0.94
computeProposers - vc 250000 20.771 ms/op 22.131 ms/op 0.94
computeEpochShuffling - vc 250000 186.64 ms/op 207.11 ms/op 0.90
getNextSyncCommittee - vc 250000 339.38 ms/op 364.91 ms/op 0.93
altair processAttestation - 250000 vs - 7PWei normalcase 32.090 ms/op 26.003 ms/op 1.23
altair processAttestation - 250000 vs - 7PWei worstcase 36.525 ms/op 30.458 ms/op 1.20
altair processAttestation - setStatus - 1/6 committees join 11.912 ms/op 9.8285 ms/op 1.21
altair processAttestation - setStatus - 1/3 committees join 23.966 ms/op 20.409 ms/op 1.17
altair processAttestation - setStatus - 1/2 committees join 36.957 ms/op 31.531 ms/op 1.17
altair processAttestation - setStatus - 2/3 committees join 48.307 ms/op 37.182 ms/op 1.30
altair processAttestation - setStatus - 4/5 committees join 60.920 ms/op 43.926 ms/op 1.39
altair processAttestation - setStatus - 100% committees join 67.082 ms/op 56.514 ms/op 1.19
altair processAttestation - updateEpochParticipants - 1/6 committees join 11.476 ms/op 10.352 ms/op 1.11
altair processAttestation - updateEpochParticipants - 1/3 committees join 23.336 ms/op 19.986 ms/op 1.17
altair processAttestation - updateEpochParticipants - 1/2 committees join 20.434 ms/op 16.922 ms/op 1.21
altair processAttestation - updateEpochParticipants - 2/3 committees join 22.587 ms/op 18.164 ms/op 1.24
altair processAttestation - updateEpochParticipants - 4/5 committees join 37.234 ms/op 20.637 ms/op 1.80
altair processAttestation - updateEpochParticipants - 100% committees join 21.550 ms/op 19.488 ms/op 1.11
altair processAttestation - updateAllStatus 17.881 ms/op 17.709 ms/op 1.01
altair processBlock - 250000 vs - 7PWei normalcase 33.737 ms/op 27.237 ms/op 1.24
altair processBlock - 250000 vs - 7PWei worstcase 101.74 ms/op 100.71 ms/op 1.01
altair processEpoch - mainnet_e81889 774.15 ms/op 754.42 ms/op 1.03
mainnet_e81889 - altair beforeProcessEpoch 318.42 ms/op 241.75 ms/op 1.32
mainnet_e81889 - altair processJustificationAndFinalization 49.948 us/op 49.261 us/op 1.01
mainnet_e81889 - altair processInactivityUpdates 17.360 ms/op 15.213 ms/op 1.14
mainnet_e81889 - altair processRewardsAndPenalties 91.922 ms/op 116.02 ms/op 0.79
mainnet_e81889 - altair processRegistryUpdates 5.4480 us/op 4.3980 us/op 1.24
mainnet_e81889 - altair processSlashings 1.0210 us/op 820.00 ns/op 1.25
mainnet_e81889 - altair processEth1DataReset 1.1370 us/op 841.00 ns/op 1.35
mainnet_e81889 - altair processEffectiveBalanceUpdates 6.5171 ms/op 5.5572 ms/op 1.17
mainnet_e81889 - altair processSlashingsReset 8.2020 us/op 8.8390 us/op 0.93
mainnet_e81889 - altair processRandaoMixesReset 11.924 us/op 19.152 us/op 0.62
mainnet_e81889 - altair processHistoricalRootsUpdate 1.2680 us/op 1.5130 us/op 0.84
mainnet_e81889 - altair processParticipationFlagUpdates 69.368 ms/op 69.044 ms/op 1.00
mainnet_e81889 - altair processSyncCommitteeUpdates 1.0930 us/op 1.0130 us/op 1.08
mainnet_e81889 - altair afterProcessEpoch 225.41 ms/op 248.10 ms/op 0.91
altair processInactivityUpdates - 250000 normalcase 73.513 ms/op 71.313 ms/op 1.03
altair processInactivityUpdates - 250000 worstcase 71.278 ms/op 60.908 ms/op 1.17
altair processParticipationFlagUpdates - 250000 anycase 54.652 ms/op 56.007 ms/op 0.98
altair processRewardsAndPenalties - 250000 normalcase 83.733 ms/op 73.039 ms/op 1.15
altair processRewardsAndPenalties - 250000 worstcase 77.757 ms/op 78.609 ms/op 0.99
altair processSyncCommitteeUpdates - 250000 359.19 ms/op 383.61 ms/op 0.94
Tree 40 250000 create 671.42 ms/op 602.31 ms/op 1.11
Tree 40 250000 get(125000) 326.56 ns/op 269.80 ns/op 1.21
Tree 40 250000 set(125000) 2.1572 us/op 1.8463 us/op 1.17
Tree 40 250000 toArray() 40.087 ms/op 33.469 ms/op 1.20
Tree 40 250000 iterate all - toArray() + loop 39.121 ms/op 34.059 ms/op 1.15
Tree 40 250000 iterate all - get(i) 120.62 ms/op 114.93 ms/op 1.05
MutableVector 250000 create 21.295 ms/op 17.299 ms/op 1.23
MutableVector 250000 get(125000) 13.176 ns/op 11.727 ns/op 1.12
MutableVector 250000 set(125000) 549.56 ns/op 466.46 ns/op 1.18
MutableVector 250000 toArray() 8.8059 ms/op 7.3043 ms/op 1.21
MutableVector 250000 iterate all - toArray() + loop 9.2883 ms/op 7.3771 ms/op 1.26
MutableVector 250000 iterate all - get(i) 3.5829 ms/op 2.7313 ms/op 1.31
Array 250000 create 5.8072 ms/op 4.6729 ms/op 1.24
Array 250000 clone - spread 2.4236 ms/op 2.1678 ms/op 1.12
Array 250000 get(125000) 1.1970 ns/op 1.2140 ns/op 0.99
Array 250000 set(125000) 1.1320 ns/op 1.2160 ns/op 0.93
Array 250000 iterate all - loop 167.97 us/op 164.72 us/op 1.02
effectiveBalanceIncrements clone Uint8Array 300000 85.478 us/op 173.18 us/op 0.49
effectiveBalanceIncrements clone MutableVector 300000 516.00 ns/op 547.00 ns/op 0.94
effectiveBalanceIncrements rw all Uint8Array 300000 301.89 us/op 182.10 us/op 1.66
effectiveBalanceIncrements rw all MutableVector 300000 169.14 ms/op 128.60 ms/op 1.32
aggregationBits - 2048 els - readonlyValues 179.01 us/op 150.65 us/op 1.19
aggregationBits - 2048 els - zipIndexesInBitList 32.416 us/op 25.940 us/op 1.25
regular array get 100000 times 67.431 us/op 65.814 us/op 1.02
wrappedArray get 100000 times 67.434 us/op 61.980 us/op 1.09
arrayWithProxy get 100000 times 32.354 ms/op 33.526 ms/op 0.97
ssz.Root.equals 1.0650 us/op 1.2450 us/op 0.86
ssz.Root.equals with valueOf() 1.3020 us/op 1.3970 us/op 0.93
byteArrayEquals with valueOf() 1.2810 us/op 1.3850 us/op 0.92
phase0 processBlock - 250000 vs - 7PWei normalcase 7.7742 ms/op 7.2352 ms/op 1.07
phase0 processBlock - 250000 vs - 7PWei worstcase 72.452 ms/op 76.173 ms/op 0.95
phase0 afterProcessEpoch - 250000 vs - 7PWei 207.75 ms/op 228.38 ms/op 0.91
phase0 beforeProcessEpoch - 250000 vs - 7PWei 535.26 ms/op 431.55 ms/op 1.24
phase0 processEpoch - mainnet_e58758 750.42 ms/op 690.91 ms/op 1.09
mainnet_e58758 - phase0 beforeProcessEpoch 419.28 ms/op 336.32 ms/op 1.25
mainnet_e58758 - phase0 processJustificationAndFinalization 48.238 us/op 48.351 us/op 1.00
mainnet_e58758 - phase0 processRewardsAndPenalties 79.380 ms/op 82.448 ms/op 0.96
mainnet_e58758 - phase0 processRegistryUpdates 34.129 us/op 29.153 us/op 1.17
mainnet_e58758 - phase0 processSlashings 966.00 ns/op 813.00 ns/op 1.19
mainnet_e58758 - phase0 processEth1DataReset 1.0740 us/op 879.00 ns/op 1.22
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 5.3446 ms/op 4.5278 ms/op 1.18
mainnet_e58758 - phase0 processSlashingsReset 7.1530 us/op 5.4610 us/op 1.31
mainnet_e58758 - phase0 processRandaoMixesReset 10.558 us/op 8.2960 us/op 1.27
mainnet_e58758 - phase0 processHistoricalRootsUpdate 1.1840 us/op 947.00 ns/op 1.25
mainnet_e58758 - phase0 processParticipationRecordUpdates 7.9170 us/op 6.6000 us/op 1.20
mainnet_e58758 - phase0 afterProcessEpoch 184.51 ms/op 203.52 ms/op 0.91
phase0 processEffectiveBalanceUpdates - 250000 normalcase 6.2212 ms/op 5.3503 ms/op 1.16
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 6.5311 ms/op 5.4759 ms/op 1.19
phase0 processRegistryUpdates - 250000 normalcase 34.724 us/op 47.590 us/op 0.73
phase0 processRegistryUpdates - 250000 badcase_full_deposits 2.9427 ms/op 2.9576 ms/op 0.99
phase0 processRegistryUpdates - 250000 worstcase 0.5 1.6211 s/op 1.3993 s/op 1.16
phase0 getAttestationDeltas - 250000 normalcase 12.959 ms/op 11.763 ms/op 1.10
phase0 getAttestationDeltas - 250000 worstcase 12.730 ms/op 11.981 ms/op 1.06
phase0 processSlashings - 250000 worstcase 34.665 ms/op 29.194 ms/op 1.19
shuffle list - 16384 els 13.233 ms/op 14.195 ms/op 0.93
shuffle list - 250000 els 188.05 ms/op 206.63 ms/op 0.91
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 455.61 us/op 461.74 us/op 0.99
pass gossip attestations to forkchoice per slot 15.020 ms/op 13.775 ms/op 1.09
computeDeltas 3.1710 ms/op 3.2009 ms/op 0.99
computeProposerBoostScoreFromBalances 503.17 us/op 452.30 us/op 1.11
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 2.3093 ms/op 1.8892 ms/op 1.22
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 753.95 us/op 585.58 us/op 1.29
BLS verify - blst-native 1.8605 ms/op 2.1835 ms/op 0.85
BLS verifyMultipleSignatures 3 - blst-native 3.8222 ms/op 4.5109 ms/op 0.85
BLS verifyMultipleSignatures 8 - blst-native 8.2439 ms/op 9.7692 ms/op 0.84
BLS verifyMultipleSignatures 32 - blst-native 30.188 ms/op 35.516 ms/op 0.85
BLS aggregatePubkeys 32 - blst-native 40.341 us/op 46.471 us/op 0.87
BLS aggregatePubkeys 128 - blst-native 154.37 us/op 181.45 us/op 0.85
getAttestationsForBlock 59.735 ms/op 59.595 ms/op 1.00
CheckpointStateCache - add get delete 18.543 us/op 17.053 us/op 1.09
validate gossip signedAggregateAndProof - struct 4.4549 ms/op 5.3807 ms/op 0.83
validate gossip signedAggregateAndProof - treeBacked 4.4667 ms/op 5.1572 ms/op 0.87
validate gossip attestation - struct 2.0822 ms/op 2.4447 ms/op 0.85
validate gossip attestation - treeBacked 2.1061 ms/op 2.4431 ms/op 0.86
pickEth1Vote - no votes 9.7737 ms/op 7.7449 ms/op 1.26
pickEth1Vote - max votes 52.073 ms/op 42.542 ms/op 1.22
pickEth1Vote - Eth1Data hashTreeRoot value x2048 24.780 ms/op 27.413 ms/op 0.90
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 9.5378 ms/op 10.169 ms/op 0.94
pickEth1Vote - Eth1Data fastSerialize value x2048 5.3924 ms/op 4.3419 ms/op 1.24
pickEth1Vote - Eth1Data fastSerialize tree x2048 24.827 ms/op 20.748 ms/op 1.20
bytes32 toHexString 1.8140 us/op 1.7260 us/op 1.05
bytes32 Buffer.toString(hex) 700.00 ns/op 759.00 ns/op 0.92
bytes32 Buffer.toString(hex) from Uint8Array 981.00 ns/op 1.0260 us/op 0.96
bytes32 Buffer.toString(hex) + 0x 705.00 ns/op 755.00 ns/op 0.93
Object access 1 prop 0.35600 ns/op 0.39700 ns/op 0.90
Map access 1 prop 0.31000 ns/op 0.34500 ns/op 0.90
Object get x1000 17.773 ns/op 11.309 ns/op 1.57
Map get x1000 1.0670 ns/op 0.97400 ns/op 1.10
Object set x1000 108.90 ns/op 71.956 ns/op 1.51
Map set x1000 69.239 ns/op 47.671 ns/op 1.45
Return object 10000 times 0.37230 ns/op 0.44730 ns/op 0.83
Throw Error 10000 times 5.7958 us/op 6.0231 us/op 0.96
enrSubnets - fastDeserialize 64 bits 1.3460 us/op 1.3900 us/op 0.97
enrSubnets - ssz BitVector 64 bits 16.454 us/op 17.392 us/op 0.95
enrSubnets - fastDeserialize 4 bits 477.00 ns/op 493.00 ns/op 0.97
enrSubnets - ssz BitVector 4 bits 2.9030 us/op 3.0390 us/op 0.96
RateTracker 1000000 limit, 1 obj count per request 182.41 ns/op 175.32 ns/op 1.04
RateTracker 1000000 limit, 2 obj count per request 136.60 ns/op 127.79 ns/op 1.07
RateTracker 1000000 limit, 4 obj count per request 115.25 ns/op 105.80 ns/op 1.09
RateTracker 1000000 limit, 8 obj count per request 102.25 ns/op 93.641 ns/op 1.09
RateTracker with prune 3.9970 us/op 3.6550 us/op 1.09
array of 16000 items push then shift 3.1591 us/op 51.572 us/op 0.06
LinkedList of 16000 items push then shift 17.205 ns/op 11.606 ns/op 1.48
array of 16000 items push then pop 201.65 ns/op 176.17 ns/op 1.14
LinkedList of 16000 items push then pop 16.171 ns/op 11.188 ns/op 1.45
array of 24000 items push then shift 4.5556 us/op 77.337 us/op 0.06
LinkedList of 24000 items push then shift 19.680 ns/op 11.861 ns/op 1.66
array of 24000 items push then pop 195.02 ns/op 173.30 ns/op 1.13
LinkedList of 24000 items push then pop 17.869 ns/op 11.255 ns/op 1.59

by benchmarkbot/action

twoeths
twoeths reviewed Mar 1, 2022
View reviewed changes
yarn.lock
@@ -2363,7 +2363,7 @@
resolved "https://registry.yarnpkg.com/@types/json5/-/json5-0.0.29.tgz#ee28707ae94e11d2b827bcbe5270bcea7f3e71ee"
integrity sha1-7ihweulOEdK4J7y+UnC86n8+ce4=

"@types/jwt-simple@^0.5.33":
"@types/[email protected]":
Copy link
Contributor

@twoeths twoeths Mar 1, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure why this is changed modified

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR #3777 did not update the yarn.lock after requiring a different version. Running yarn causes this diff

@wemeetagain wemeetagain merged commit e7442db into master Mar 1, 2022
@wemeetagain wemeetagain deleted the dapplion/unstable-benchmark branch March 1, 2022 15:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@twoeths twoeths twoeths approved these changes

@wemeetagain wemeetagain wemeetagain approved these changes

@g11tech g11tech Awaiting requested review from g11tech

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@dapplion @wemeetagain @twoeths