Bring reference_hardware.json inline with machine used for weights #5196
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Alexandru Gheorghe <[email protected]>
Signed-off-by: Alexandru Gheorghe <[email protected]>
AndreiEres
reviewed
Jul 31, 2024
substrate/utils/frame/benchmarking-cli/src/machine/reference_hardware.json
Outdated
Show resolved
Hide resolved
|
The CI pipeline was cancelled due to failure one of the required jobs. |
Signed-off-by: Alexandru Gheorghe <[email protected]>
Signed-off-by: Alexandru Gheorghe <[email protected]>
ggwpez
approved these changes
Jul 31, 2024
1 task
koute
approved these changes
Aug 5, 2024
ordian
added a commit
that referenced
this pull request
Aug 7, 2024
* master: Umbrella crate: exclude chain-specific crates (#5173) Bring reference_hardware.json inline with machine used for weights (#5196) Snowbridge on Westend (#5074) Run semver check even when no prdoc (#5189) Export more from sc-service (#5250) Update the wishlist leaderboard script to handle PRs (#5256)
btwiuse
pushed a commit
to btwiuse/substrate-benchmark-machine
that referenced
this pull request
Aug 9, 2024
dharjeezy
pushed a commit
to dharjeezy/polkadot-sdk
that referenced
this pull request
Aug 28, 2024
…aritytech#5196) Since `May 2023` after paritytech/substrate#13548 optimization, `Blake2256` is faster with about 30%, that means that there is a difference of ~30% between the benchmark values we ask validators to run against and the machine we use for generating the weights.So if all validators, just barely pass the benchmarks our weights are potentially underestimated with about ~20%, so let's bring this two in sync. Same thing happened when we merged paritytech#2524 in `Nov 2023` SR25519-Verify became faster with about 10-15% ## Results Generated on machine from here: https://github.com/paritytech/devops/pull/3210 ``` +----------+----------------+--------------+-------------+-------------------+ | Category | Function | Score | Minimum | Result | +============================================================================+ | CPU | BLAKE2-256 | 1.00 GiBs | 783.27 MiBs | ✅ Pass (130.7 %) | |----------+----------------+--------------+-------------+-------------------| | CPU | SR25519-Verify | 637.62 KiBs | 560.67 KiBs | ✅ Pass (113.7 %) | |----------+----------------+--------------+-------------+-------------------| | Memory | Copy | 12.19 GiBs | 11.49 GiBs | ✅ Pass (106.1 %) | ``` Discovered and discussed here: paritytech#5127 (comment) ## Downsides Machines that barely passed the benchmark will suddenly find themselves bellow the benchmark, but since that is just an warning and everything else continues as before it shouldn't be too impactful and should give the validators the necessary information that they need to become compliant, since they actually aren't when compared with the used weights. --------- Signed-off-by: Alexandru Gheorghe <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Since
May 2023after paritytech/substrate#13548 optimization,Blake2256is faster with about 30%, that means that there is a difference of ~30% between the benchmark values we ask validators to run against and the machine we use for generating the weights.So if all validators, just barely pass the benchmarks our weights are potentially underestimated with about ~20%, so let's bring this two in sync.Same thing happened when we merged #2524 in
Nov 2023SR25519-Verify became faster with about 10-15%Results
Generated on machine from here: https://github.com/paritytech/devops/pull/3210
Discovered and discussed here: #5127 (comment)
Downsides
Machines that barely passed the benchmark will suddenly find themselves bellow the benchmark, but since that is just an warning and everything else continues as before it shouldn't be too impactful and should give the validators the necessary information that they need to become compliant, since they actually aren't when compared with the used weights.