Per runtime benchmarks #490

girazoki · 2024-04-09T14:07:56Z

Adds specific benchmarks for each of the runtimes we host: flashbox, dancebox, simple-template and frontier-template. In order to do this:

I added a new benchmarking template for runtimes. This one is different from the pallet one.
I added new arguments to our benchmarking tool. I will write a guide explaining how to use it.
I had to fix several parts of the benchmarking code that was broken

As I advanced on this, I realized that our frontier template was not applying the 1e6 multiplier to all deposits/fees. I also fixed this plus adapted tests.

MIssing:

Add complete guide on how to benchmark pallets & runtimes
Fix request_delegate benchmark (One db read missing)
Wire frame-system
Benchmark again in benchmarking server
Fix XCM
Cleanup

…e-benchs

github-actions · 2024-04-09T17:57:26Z

Coverage Report

(master)

@@                       Coverage Diff                       @@
##           master   girazoki-per-runtime-benchs      +/-   ##
===============================================================
- Coverage   65.38%                        65.36%   -0.02%     
  Files          68                            68              
+ Lines       10000                         10001       +1     
===============================================================
- Hits         6538                          6537       -1     
+ Misses       3462                          3464       +2

Files Changed	Coverage
/pallets/invulnerables/src/lib.rs	67.89% (-0.63%)	🔽
/pallets/invulnerables/src/weights.rs	25.56% (-1.11%)	🔽

Coverage generated Thu Apr 18 11:26:16 UTC 2024

pallets/data-preservers/src/lib.rs

…e-benchs

benchmarking/frame-weight-runtime-template.hbs

…e-benchs

fgamundi

Looks good. Nice doc!

docs/benchmarking.md

…e-benchs

tmpolaczyk · 2024-04-17T16:08:17Z

Using rg 'weights::', I can see some pallets are not using the per runtime weights, is this expected?

pallet_session and pallet_treasury in dancebox, many pallets in flashbox, and some pallets in templates:

container-chains/templates/simple/runtime/src/lib.rs
415:    type WeightInfo = cumulus_pallet_parachain_system::weights::SubstrateWeight<Runtime>;
614:    type TimestampWeights = pallet_timestamp::weights::SubstrateWeight<Runtime>;
615:    type AuthorInherentWeights = pallet_author_inherent::weights::SubstrateWeight<Runtime>;
616:    type AuthoritiesNotingWeights = pallet_cc_authorities_noting::weights::SubstrateWeight<Runtime>;

container-chains/templates/frontier/runtime/src/lib.rs
807:    type WeightInfo = pallet_hotfix_sufficients::weights::SubstrateWeight<Runtime>;
826:    type TimestampWeights = pallet_timestamp::weights::SubstrateWeight<Runtime>;
827:    type AuthorInherentWeights = pallet_author_inherent::weights::SubstrateWeight<Runtime>;
828:    type AuthoritiesNotingWeights = pallet_cc_authorities_noting::weights::SubstrateWeight<Runtime>;

runtime/dancebox/src/lib.rs
708:    type WeightInfo = pallet_session::weights::SubstrateWeight<Runtime>;
1602:    type WeightInfo = pallet_treasury::weights::SubstrateWeight<Runtime>;


runtime/flashbox/src/lib.rs
52:        weights::{
339:    type SystemWeightInfo = weights::frame_system::SubstrateWeight<Runtime>;
360:    type WeightInfo = weights::pallet_timestamp::SubstrateWeight<Runtime>;
392:    type WeightInfo = weights::pallet_author_inherent::SubstrateWeight<Runtime>;
415:    type WeightInfo = weights::pallet_balances::SubstrateWeight<Runtime>;
481:    type WeightInfo = weights::cumulus_pallet_parachain_system::SubstrateWeight<Runtime>;
602:    type WeightInfo = pallet_session::weights::SubstrateWeight<Runtime>;
733:    type WeightInfo = weights::pallet_collator_assignment::SubstrateWeight<Runtime>;
783:    type WeightInfo = pallet_services_payment::weights::SubstrateWeight<Runtime>;
792:    type WeightInfo = pallet_data_preservers::weights::SubstrateWeight<Runtime>;
807:    type WeightInfo = pallet_author_noting::weights::SubstrateWeight<Runtime>;
826:    type WeightInfo = pallet_invulnerables::weights::SubstrateWeight<Runtime>;
852:    type WeightInfo = pallet_configuration::weights::SubstrateWeight<Runtime>;
917:    type WeightInfo = pallet_registrar::weights::SubstrateWeight<Runtime>;
929:    type WeightInfo = pallet_sudo::weights::SubstrateWeight<Runtime>;
936:    type WeightInfo = pallet_utility::weights::SubstrateWeight<Runtime>;
1046:    type WeightInfo = pallet_proxy::weights::SubstrateWeight<Runtime>;
1164:    type WeightInfo = weights::pallet_tx_pause::SubstrateWeight<Runtime>;
1305:    type WeightInfo = weights::pallet_stream_payment::SubstrateWeight<Runtime>;
1339:    type WeightInfo = weights::pallet_identity::SubstrateWeight<Runtime>;
1364:    type WeightInfo = pallet_treasury::weights::SubstrateWeight<Runtime>;
1397:    type WeightInfo = weights::pallet_multisig::SubstrateWeight<Runtime>;

ParthDesai · 2024-04-17T17:40:14Z

container-chains/templates/frontier/runtime/src/lib.rs

+                    // inject it into pallet-foreign-asset-creator.
+                    let asset_location = MultiLocation::new(
+                        0,
+                        X2(PalletInstance(50), GeneralIndex(u32::from(asset_id).into()))


Can we define some constant and use it? That would ensure nobody accidentally modify this.

I will look into this

would you want this to be somewhere common to all runtimes? If so, I think this would leave in dancekit, and I would need to cherry-pick the change here. Maybe it's a cleanup we can do in a follow up?

My idea actually is that we generate in foreign asset creator a benchmarking code that is called: register_foreign_asset_for_benchmarking which would abstract all this. But again I think we can leave this for future PR

ParthDesai · 2024-04-18T06:54:09Z

How does benchmark of a pallet as part of runtime and standalone differs? Apart from the configuration difference actual code is more or less same.

ParthDesai · 2024-04-18T06:56:03Z

Using rg 'weights::', I can see some pallets are not using the per runtime weights, is this expected?

pallet_session and pallet_treasury in dancebox, many pallets in flashbox, and some pallets in templates:

container-chains/templates/simple/runtime/src/lib.rs
415:    type WeightInfo = cumulus_pallet_parachain_system::weights::SubstrateWeight<Runtime>;
614:    type TimestampWeights = pallet_timestamp::weights::SubstrateWeight<Runtime>;
615:    type AuthorInherentWeights = pallet_author_inherent::weights::SubstrateWeight<Runtime>;
616:    type AuthoritiesNotingWeights = pallet_cc_authorities_noting::weights::SubstrateWeight<Runtime>;

container-chains/templates/frontier/runtime/src/lib.rs
807:    type WeightInfo = pallet_hotfix_sufficients::weights::SubstrateWeight<Runtime>;
826:    type TimestampWeights = pallet_timestamp::weights::SubstrateWeight<Runtime>;
827:    type AuthorInherentWeights = pallet_author_inherent::weights::SubstrateWeight<Runtime>;
828:    type AuthoritiesNotingWeights = pallet_cc_authorities_noting::weights::SubstrateWeight<Runtime>;

runtime/dancebox/src/lib.rs
708:    type WeightInfo = pallet_session::weights::SubstrateWeight<Runtime>;
1602:    type WeightInfo = pallet_treasury::weights::SubstrateWeight<Runtime>;


runtime/flashbox/src/lib.rs
52:        weights::{
339:    type SystemWeightInfo = weights::frame_system::SubstrateWeight<Runtime>;
360:    type WeightInfo = weights::pallet_timestamp::SubstrateWeight<Runtime>;
392:    type WeightInfo = weights::pallet_author_inherent::SubstrateWeight<Runtime>;
415:    type WeightInfo = weights::pallet_balances::SubstrateWeight<Runtime>;
481:    type WeightInfo = weights::cumulus_pallet_parachain_system::SubstrateWeight<Runtime>;
602:    type WeightInfo = pallet_session::weights::SubstrateWeight<Runtime>;
733:    type WeightInfo = weights::pallet_collator_assignment::SubstrateWeight<Runtime>;
783:    type WeightInfo = pallet_services_payment::weights::SubstrateWeight<Runtime>;
792:    type WeightInfo = pallet_data_preservers::weights::SubstrateWeight<Runtime>;
807:    type WeightInfo = pallet_author_noting::weights::SubstrateWeight<Runtime>;
826:    type WeightInfo = pallet_invulnerables::weights::SubstrateWeight<Runtime>;
852:    type WeightInfo = pallet_configuration::weights::SubstrateWeight<Runtime>;
917:    type WeightInfo = pallet_registrar::weights::SubstrateWeight<Runtime>;
929:    type WeightInfo = pallet_sudo::weights::SubstrateWeight<Runtime>;
936:    type WeightInfo = pallet_utility::weights::SubstrateWeight<Runtime>;
1046:    type WeightInfo = pallet_proxy::weights::SubstrateWeight<Runtime>;
1164:    type WeightInfo = weights::pallet_tx_pause::SubstrateWeight<Runtime>;
1305:    type WeightInfo = weights::pallet_stream_payment::SubstrateWeight<Runtime>;
1339:    type WeightInfo = weights::pallet_identity::SubstrateWeight<Runtime>;
1364:    type WeightInfo = pallet_treasury::weights::SubstrateWeight<Runtime>;
1397:    type WeightInfo = weights::pallet_multisig::SubstrateWeight<Runtime>;

+1

girazoki · 2024-04-18T07:29:22Z

Using rg 'weights::', I can see some pallets are not using the per runtime weights, is this expected?

pallet_session and pallet_treasury in dancebox, many pallets in flashbox, and some pallets in templates:

container-chains/templates/simple/runtime/src/lib.rs
415:    type WeightInfo = cumulus_pallet_parachain_system::weights::SubstrateWeight<Runtime>;
614:    type TimestampWeights = pallet_timestamp::weights::SubstrateWeight<Runtime>;
615:    type AuthorInherentWeights = pallet_author_inherent::weights::SubstrateWeight<Runtime>;
616:    type AuthoritiesNotingWeights = pallet_cc_authorities_noting::weights::SubstrateWeight<Runtime>;

container-chains/templates/frontier/runtime/src/lib.rs
807:    type WeightInfo = pallet_hotfix_sufficients::weights::SubstrateWeight<Runtime>;
826:    type TimestampWeights = pallet_timestamp::weights::SubstrateWeight<Runtime>;
827:    type AuthorInherentWeights = pallet_author_inherent::weights::SubstrateWeight<Runtime>;
828:    type AuthoritiesNotingWeights = pallet_cc_authorities_noting::weights::SubstrateWeight<Runtime>;

runtime/dancebox/src/lib.rs
708:    type WeightInfo = pallet_session::weights::SubstrateWeight<Runtime>;
1602:    type WeightInfo = pallet_treasury::weights::SubstrateWeight<Runtime>;


runtime/flashbox/src/lib.rs
52:        weights::{
339:    type SystemWeightInfo = weights::frame_system::SubstrateWeight<Runtime>;
360:    type WeightInfo = weights::pallet_timestamp::SubstrateWeight<Runtime>;
392:    type WeightInfo = weights::pallet_author_inherent::SubstrateWeight<Runtime>;
415:    type WeightInfo = weights::pallet_balances::SubstrateWeight<Runtime>;
481:    type WeightInfo = weights::cumulus_pallet_parachain_system::SubstrateWeight<Runtime>;
602:    type WeightInfo = pallet_session::weights::SubstrateWeight<Runtime>;
733:    type WeightInfo = weights::pallet_collator_assignment::SubstrateWeight<Runtime>;
783:    type WeightInfo = pallet_services_payment::weights::SubstrateWeight<Runtime>;
792:    type WeightInfo = pallet_data_preservers::weights::SubstrateWeight<Runtime>;
807:    type WeightInfo = pallet_author_noting::weights::SubstrateWeight<Runtime>;
826:    type WeightInfo = pallet_invulnerables::weights::SubstrateWeight<Runtime>;
852:    type WeightInfo = pallet_configuration::weights::SubstrateWeight<Runtime>;
917:    type WeightInfo = pallet_registrar::weights::SubstrateWeight<Runtime>;
929:    type WeightInfo = pallet_sudo::weights::SubstrateWeight<Runtime>;
936:    type WeightInfo = pallet_utility::weights::SubstrateWeight<Runtime>;
1046:    type WeightInfo = pallet_proxy::weights::SubstrateWeight<Runtime>;
1164:    type WeightInfo = weights::pallet_tx_pause::SubstrateWeight<Runtime>;
1305:    type WeightInfo = weights::pallet_stream_payment::SubstrateWeight<Runtime>;
1339:    type WeightInfo = weights::pallet_identity::SubstrateWeight<Runtime>;
1364:    type WeightInfo = pallet_treasury::weights::SubstrateWeight<Runtime>;
1397:    type WeightInfo = weights::pallet_multisig::SubstrateWeight<Runtime>;

For pallet-session, there is really no benchmark we can do for now. The benchmarking code depends on pallet-session-historical and pallet-staking that we dont have. For the rest that you mention, I will take a look, it's probably that I forgot to add them.

For pallet-hotfix-sufficients, I believe this is a pallet we should get rid of as it was introduced in moonbeam to fix a bug a long time ago. So it does not really matter if it uses official weights or not I would say, as it is going to dissaper

girazoki · 2024-04-18T07:55:20Z

How does benchmark of a pallet as part of runtime and standalone differs? Apart from the configuration difference actual code is more or less same.

Different runtimes might have different hooks for the same pallet. The benchmarking code is the same, but the surrounding conditions are different. And for each runtime, we can have different weights associated.

What we call the "standalone" pallet benchmarking is basically a benchmarking that we dont trully care (as long as it's not generated against a mocked runtime obviously) against which runtime is used, nor if it is generated in the benchmarking server. It's only to generate the WeightInfo trait and some reference weights. Tipically I benchmark "standalone pallets" against dancebox, but this is just a choice.

girazoki added 16 commits April 4, 2024 17:11

modify author-noting hook for runtime benches

3f3cfae

fix collator-assignment-bench

d8cba86

add missing benchmarks and order

56528e5

fix flashbox

aac2765

start generating

48bba50

wip

caab1b2

Everything compiling already

d5b9385

fix benchmarks in frontier templ

b37b039

select a different template to benchmark runtimes

b95b3ab

fixing more benches

31a4e24

add missing weights and modify reward_invulnerable

794ecb0

fix xcm bench

be79075

fix xcm bench

0df75d3

pin to latest moonkit

4780245

Merge remote-tracking branch 'origin/master' into girazoki-per-runtim…

c7c6f82

…e-benchs

more changes

e409c60

girazoki added 3 commits April 10, 2024 14:28

fix leftover

2988998

wip

03a36e6

fixes

93dd73d

girazoki added D9-needsaudit👮 PR contains changes to fund-managing logic that should be properly reviewed and externally audited B7-runtimenoteworthy Changes should be noted in any runtime-upgrade release notes breaking Needs to be mentioned in breaking changes labels Apr 10, 2024

finally I think all weights working

083fe8d

girazoki commented Apr 10, 2024

View reviewed changes

pallets/data-preservers/src/lib.rs Outdated Show resolved Hide resolved

girazoki added 5 commits April 10, 2024 19:27

adapt to latest dancekit

a9e27e2

Merge remote-tracking branch 'origin/master' into girazoki-per-runtim…

1804e19

…e-benchs

fix weights

76a27b1

start fixing tests

cc61231

fix rust tests

205a60b

girazoki requested a review from ParthDesai April 12, 2024 12:02

girazoki added 3 commits April 15, 2024 11:29

benchmarked weights

e1840ed

support xcm

231fdac

xcm weights fixed

aadabe7

tmpolaczyk reviewed Apr 15, 2024

View reviewed changes

benchmarking/frame-weight-runtime-template.hbs Outdated Show resolved Hide resolved

girazoki added 6 commits April 15, 2024 14:44

remove if

7c55c55

add xcm benchmarks for containrs

59b167d

good xcm benchmarks

e5effc5

Merge remote-tracking branch 'origin/master' into girazoki-per-runtim…

d6cbb34

…e-benchs

update lock

ded1dee

continue with fixes

e3ddf6c

fgamundi approved these changes Apr 17, 2024

View reviewed changes

docs/benchmarking.md Outdated Show resolved Hide resolved

girazoki added 2 commits April 17, 2024 16:36

Merge remote-tracking branch 'origin/master' into girazoki-per-runtim…

781406e

…e-benchs

update typo

211d4e1

ParthDesai reviewed Apr 17, 2024

View reviewed changes

girazoki added 2 commits April 18, 2024 09:35

fix weights

dbb07af

fix build

9238111

fix set_latest_authorities wights

7a720b6

ParthDesai approved these changes Apr 18, 2024

View reviewed changes

girazoki added 4 commits April 18, 2024 11:47

last pieces

b20f9a3

pallet session benches

f1fee83

add it to flashbox too

a1bdd67

Add pallet session weights2

f83ea03

girazoki merged commit faf9e75 into master Apr 18, 2024
32 checks passed

girazoki deleted the girazoki-per-runtime-benchs branch April 18, 2024 11:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Per runtime benchmarks #490

Per runtime benchmarks #490

girazoki commented Apr 9, 2024 •

edited

Loading

github-actions bot commented Apr 9, 2024 •

edited

Loading

fgamundi left a comment

tmpolaczyk commented Apr 17, 2024 •

edited

Loading

ParthDesai Apr 17, 2024

girazoki Apr 18, 2024

girazoki Apr 18, 2024

girazoki Apr 18, 2024

ParthDesai commented Apr 18, 2024

ParthDesai commented Apr 18, 2024

girazoki commented Apr 18, 2024 •

edited

Loading

girazoki commented Apr 18, 2024

Per runtime benchmarks #490

Per runtime benchmarks #490

Conversation

girazoki commented Apr 9, 2024 • edited Loading

github-actions bot commented Apr 9, 2024 • edited Loading

Coverage Report

fgamundi left a comment

Choose a reason for hiding this comment

tmpolaczyk commented Apr 17, 2024 • edited Loading

ParthDesai Apr 17, 2024

Choose a reason for hiding this comment

girazoki Apr 18, 2024

Choose a reason for hiding this comment

girazoki Apr 18, 2024

Choose a reason for hiding this comment

girazoki Apr 18, 2024

Choose a reason for hiding this comment

ParthDesai commented Apr 18, 2024

ParthDesai commented Apr 18, 2024

girazoki commented Apr 18, 2024 • edited Loading

girazoki commented Apr 18, 2024

girazoki commented Apr 9, 2024 •

edited

Loading

github-actions bot commented Apr 9, 2024 •

edited

Loading

tmpolaczyk commented Apr 17, 2024 •

edited

Loading

girazoki commented Apr 18, 2024 •

edited

Loading