Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Contracts: Rework host fn benchmarks #4233

Merged
merged 48 commits into from
May 23, 2024
Merged

Contracts: Rework host fn benchmarks #4233

merged 48 commits into from
May 23, 2024

Conversation

pgherveou
Copy link
Contributor

@pgherveou pgherveou commented Apr 22, 2024

fix #4163

This PR does the following:
Update to pallet-contracts-proc-macro:

  • Parse #[cfg] so we can add a dummy noop host function for benchmark.
  • Generate BenchEnv::<host_fn> so we can call host functions directly in the benchmark.
  • Add the weight of the noop host function before calling the host function itself

Update benchmarks:

  • Update all host function benchmark, a host function benchmark now simply call the host function, instead of invoking the function n times from within a contract.
  • Refactor RuntimeCosts & Schedule, for most host functions, we can now use the generated weight function directly instead of computing the diff with the cost! macro
// Before
#[benchmark(pov_mode = Measured)]
fn seal_input(r: Linear<0, API_BENCHMARK_RUNS>) {
    let code = WasmModule::<T>::from(ModuleDefinition {
        memory: Some(ImportedMemory::max::<T>()),
        imported_functions: vec![ImportedFunction {
            module: "seal0",
            name: "seal_input",
            params: vec![ValueType::I32, ValueType::I32],
            return_type: None,
        }],
        data_segments: vec![DataSegment { offset: 0, value: 0u32.to_le_bytes().to_vec() }],
        call_body: Some(body::repeated(
            r,
            &[
                Instruction::I32Const(4), // ptr where to store output
                Instruction::I32Const(0), // ptr to length
                Instruction::Call(0),
            ],
        )),
        ..Default::default()
    });

    call_builder!(func, code);

    let res;
    #[block]
    {
        res = func.call();
    }
    assert_eq!(res.did_revert(), false);
}
// After
fn seal_input(n: Linear<0, { code::max_pages::<T>() * 64 * 1024 - 4 }>) {
    let mut setup = CallSetup::<T>::default();
    let (mut ext, _) = setup.ext();
    let mut runtime = crate::wasm::Runtime::new(&mut ext, vec![42u8; n as usize]);
    let mut memory = memory!(n.to_le_bytes(), vec![0u8; n as usize],);
    let result;
    #[block]
    {
        result = BenchEnv::seal0_input(&mut runtime, &mut memory, 4, 0)
    }
    assert_ok!(result);
    assert_eq!(&memory[4..], &vec![42u8; n as usize]);
}

Weights compare

@pgherveou pgherveou changed the title Contracts: Rework host fn benchmarks Contracts: Rework host fn benchmarks (wip) Apr 22, 2024
@pgherveou pgherveou marked this pull request as ready for review April 26, 2024 16:54
@pgherveou pgherveou requested a review from athei as a code owner April 26, 2024 16:54
@pgherveou
Copy link
Contributor Author

bot bench substrate-pallet --pallet=pallet_contracts

@command-bot
Copy link

command-bot bot commented Apr 29, 2024

@pgherveou https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/6079780 was started for your command "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" --subcommand=pallet --runtime=dev --target_dir=substrate --pallet=pallet_contracts. Check out https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/pipelines?page=1&scope=all&username=group_605_bot to know what else is being executed currently.

Comment bot cancel 2-7dfaefeb-354e-4724-9d02-8a1bc5c47e45 to cancel this command or bot cancel to cancel all commands in this pull request.

@command-bot
Copy link

command-bot bot commented Apr 29, 2024

@pgherveou Command "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" --subcommand=pallet --runtime=dev --target_dir=substrate --pallet=pallet_contracts has finished. Result: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/6079780 has finished. If any artifacts were generated, you can download them from https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/6079780/artifacts/download.

@pgherveou
Copy link
Contributor Author

bot bench substrate-pallet --pallet=pallet_contracts

@command-bot
Copy link

command-bot bot commented Apr 29, 2024

@pgherveou https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/6082811 was started for your command "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" --subcommand=pallet --runtime=dev --target_dir=substrate --pallet=pallet_contracts. Check out https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/pipelines?page=1&scope=all&username=group_605_bot to know what else is being executed currently.

Comment bot cancel 7-b73e1a52-9d8a-4f41-9b1d-2ca6f1dd95a3 to cancel this command or bot cancel to cancel all commands in this pull request.

@command-bot
Copy link

command-bot bot commented Apr 29, 2024

@pgherveou Command "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" --subcommand=pallet --runtime=dev --target_dir=substrate --pallet=pallet_contracts has finished. Result: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/6082811 has finished. If any artifacts were generated, you can download them from https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/6082811/artifacts/download.

@pgherveou pgherveou changed the title Contracts: Rework host fn benchmarks (wip) Contracts: Rework host fn benchmarks Apr 29, 2024
@command-bot
Copy link

command-bot bot commented May 19, 2024

@pgherveou Command "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" --subcommand=pallet --runtime=dev --target_dir=substrate --pallet=pallet_contracts has finished. Result: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/6243433 has finished. If any artifacts were generated, you can download them from https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/6243433/artifacts/download.

@paritytech-cicd-pr
Copy link

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-clippy
Logs: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/6246684

@pgherveou
Copy link
Contributor Author

bot bench substrate-pallet --pallet=pallet_contracts

@command-bot
Copy link

command-bot bot commented May 20, 2024

@pgherveou https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/6247144 was started for your command "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" --subcommand=pallet --runtime=dev --target_dir=substrate --pallet=pallet_contracts. Check out https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/pipelines?page=1&scope=all&username=group_605_bot to know what else is being executed currently.

Comment bot cancel 1-2aadf67f-754a-4d84-bd7a-5553870d15ee to cancel this command or bot cancel to cancel all commands in this pull request.

@command-bot
Copy link

command-bot bot commented May 20, 2024

@pgherveou Command "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" --subcommand=pallet --runtime=dev --target_dir=substrate --pallet=pallet_contracts has finished. Result: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/6247144 has finished. If any artifacts were generated, you can download them from https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/6247144/artifacts/download.

@pgherveou
Copy link
Contributor Author

Something still doesn't add up with the seal_instantiate benchmark. The base weight is super low (lower than seal_call) and then a lot is added in case of transfer: Screenshot 2024-05-18 at 07 31 52

@athei Actually I think it does adds up
We are just forgetting the RocksDbWeight (which confusingly are actually adding ref_time and not pov)
on seal_call they are attached to t since we only touch accounts if we do a transfer.
On instantiate it happens all the time and does not depend on t

so if you count these 3 lines on seal_call

.saturating_add(Weight::from_parts(44_066_869, 0).saturating_mul(t.into()))
.saturating_add(RocksDbWeight::get().reads((1_u64).saturating_mul(t.into())))
.saturating_add(RocksDbWeight::get().writes((1_u64).saturating_mul(t.into())))

// 44_000_000 + 25_000_000 + 100_000_000 ~=  169_000_000

You roughly get what we have in seal_instantiate

.saturating_add(Weight::from_parts(159_867_027, 0).saturating_mul(t.into()))

Copy link
Member

@athei athei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right. I wasn't accounting for the reads and writes which are added unconditionally for seal_instantiate.

Copy link
Contributor

@Robbepop Robbepop left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change is very reasonable and I am glad that it has been done. The new resulting numbers look reasonable and should be more precise than before. Thanks @pgherveou !

@pgherveou pgherveou added this pull request to the merge queue May 23, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks May 23, 2024
Copy link
Member

@xermicus xermicus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@athei athei enabled auto-merge May 23, 2024 10:53
@athei athei added this pull request to the merge queue May 23, 2024
Merged via the queue into master with commit 493ba5e May 23, 2024
148 of 151 checks passed
@athei athei deleted the pg/rework-host-benchs branch May 23, 2024 11:37
hitchhooker pushed a commit to ibp-network/polkadot-sdk that referenced this pull request Jun 5, 2024
fix paritytech#4163

This PR does the following:
Update to pallet-contracts-proc-macro: 
- Parse #[cfg] so we can add a dummy noop host function for benchmark.
- Generate BenchEnv::<host_fn> so we can call host functions directly in
the benchmark.
- Add the weight of the noop host function before calling the host
function itself

Update benchmarks:
- Update all host function benchmark, a host function benchmark now
simply call the host function, instead of invoking the function n times
from within a contract.
- Refactor RuntimeCosts & Schedule, for most host functions, we can now
use the generated weight function directly instead of computing the diff
with the cost! macro

```rust
// Before
#[benchmark(pov_mode = Measured)]
fn seal_input(r: Linear<0, API_BENCHMARK_RUNS>) {
    let code = WasmModule::<T>::from(ModuleDefinition {
        memory: Some(ImportedMemory::max::<T>()),
        imported_functions: vec![ImportedFunction {
            module: "seal0",
            name: "seal_input",
            params: vec![ValueType::I32, ValueType::I32],
            return_type: None,
        }],
        data_segments: vec![DataSegment { offset: 0, value: 0u32.to_le_bytes().to_vec() }],
        call_body: Some(body::repeated(
            r,
            &[
                Instruction::I32Const(4), // ptr where to store output
                Instruction::I32Const(0), // ptr to length
                Instruction::Call(0),
            ],
        )),
        ..Default::default()
    });

    call_builder!(func, code);

    let res;
    #[block]
    {
        res = func.call();
    }
    assert_eq!(res.did_revert(), false);
}
```

```rust
// After
fn seal_input(n: Linear<0, { code::max_pages::<T>() * 64 * 1024 - 4 }>) {
    let mut setup = CallSetup::<T>::default();
    let (mut ext, _) = setup.ext();
    let mut runtime = crate::wasm::Runtime::new(&mut ext, vec![42u8; n as usize]);
    let mut memory = memory!(n.to_le_bytes(), vec![0u8; n as usize],);
    let result;
    #[block]
    {
        result = BenchEnv::seal0_input(&mut runtime, &mut memory, 4, 0)
    }
    assert_ok!(result);
    assert_eq!(&memory[4..], &vec![42u8; n as usize]);
}
``` 

[Weights
compare](https://weights.tasty.limo/compare?unit=weight&ignore_errors=true&threshold=10&method=asymptotic&repo=polkadot-sdk&old=master&new=pg%2Frework-host-benchs&path_pattern=substrate%2Fframe%2Fcontracts%2Fsrc%2Fweights.rs%2Cpolkadot%2Fruntime%2F*%2Fsrc%2Fweights%2F**%2F*.rs%2Cpolkadot%2Fbridges%2Fmodules%2F*%2Fsrc%2Fweights.rs%2Ccumulus%2F**%2Fweights%2F*.rs%2Ccumulus%2F**%2Fweights%2Fxcm%2F*.rs%2Ccumulus%2F**%2Fsrc%2Fweights.rs)

---------

Co-authored-by: command-bot <>
Co-authored-by: Alexander Theißen <[email protected]>
TarekkMA pushed a commit to moonbeam-foundation/polkadot-sdk that referenced this pull request Aug 2, 2024
fix paritytech#4163

This PR does the following:
Update to pallet-contracts-proc-macro: 
- Parse #[cfg] so we can add a dummy noop host function for benchmark.
- Generate BenchEnv::<host_fn> so we can call host functions directly in
the benchmark.
- Add the weight of the noop host function before calling the host
function itself

Update benchmarks:
- Update all host function benchmark, a host function benchmark now
simply call the host function, instead of invoking the function n times
from within a contract.
- Refactor RuntimeCosts & Schedule, for most host functions, we can now
use the generated weight function directly instead of computing the diff
with the cost! macro

```rust
// Before
#[benchmark(pov_mode = Measured)]
fn seal_input(r: Linear<0, API_BENCHMARK_RUNS>) {
    let code = WasmModule::<T>::from(ModuleDefinition {
        memory: Some(ImportedMemory::max::<T>()),
        imported_functions: vec![ImportedFunction {
            module: "seal0",
            name: "seal_input",
            params: vec![ValueType::I32, ValueType::I32],
            return_type: None,
        }],
        data_segments: vec![DataSegment { offset: 0, value: 0u32.to_le_bytes().to_vec() }],
        call_body: Some(body::repeated(
            r,
            &[
                Instruction::I32Const(4), // ptr where to store output
                Instruction::I32Const(0), // ptr to length
                Instruction::Call(0),
            ],
        )),
        ..Default::default()
    });

    call_builder!(func, code);

    let res;
    #[block]
    {
        res = func.call();
    }
    assert_eq!(res.did_revert(), false);
}
```

```rust
// After
fn seal_input(n: Linear<0, { code::max_pages::<T>() * 64 * 1024 - 4 }>) {
    let mut setup = CallSetup::<T>::default();
    let (mut ext, _) = setup.ext();
    let mut runtime = crate::wasm::Runtime::new(&mut ext, vec![42u8; n as usize]);
    let mut memory = memory!(n.to_le_bytes(), vec![0u8; n as usize],);
    let result;
    #[block]
    {
        result = BenchEnv::seal0_input(&mut runtime, &mut memory, 4, 0)
    }
    assert_ok!(result);
    assert_eq!(&memory[4..], &vec![42u8; n as usize]);
}
``` 

[Weights
compare](https://weights.tasty.limo/compare?unit=weight&ignore_errors=true&threshold=10&method=asymptotic&repo=polkadot-sdk&old=master&new=pg%2Frework-host-benchs&path_pattern=substrate%2Fframe%2Fcontracts%2Fsrc%2Fweights.rs%2Cpolkadot%2Fruntime%2F*%2Fsrc%2Fweights%2F**%2F*.rs%2Cpolkadot%2Fbridges%2Fmodules%2F*%2Fsrc%2Fweights.rs%2Ccumulus%2F**%2Fweights%2F*.rs%2Ccumulus%2F**%2Fweights%2Fxcm%2F*.rs%2Ccumulus%2F**%2Fsrc%2Fweights.rs)

---------

Co-authored-by: command-bot <>
Co-authored-by: Alexander Theißen <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
T7-smart_contracts This PR/Issue is related to smart contracts.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

contracts: Make host function benchmarks architecture independent
6 participants