Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

subsystem-bench: Prepare CI output #3158

Merged
merged 11 commits into from
Feb 6, 2024

Conversation

AndreiEres
Copy link
Contributor

@AndreiEres AndreiEres commented Jan 31, 2024

  1. Benchmark results are collected in a single struct.
  2. The output of the results is prettified.
  3. The result struct used to save the output as a yaml and store it in artifacts in a CI job.
$ cargo run -p polkadot-subsystem-bench --release -- test-sequence --path polkadot/node/subsystem-bench/examples/availability_read.yaml | tee output.txt
$ cat output.txt

polkadot/node/subsystem-bench/examples/availability_read.yaml #1

Network usage, KiB                     total   per block
Received from peers               510796.000  170265.333
Sent to peers                        221.000      73.667

CPU usage, s                           total   per block
availability-recovery                 38.671      12.890
Test environment                       0.255       0.085


polkadot/node/subsystem-bench/examples/availability_read.yaml #2

Network usage, KiB                     total   per block
Received from peers               413633.000  137877.667
Sent to peers                        353.000     117.667

CPU usage, s                           total   per block
availability-recovery                 52.630      17.543
Test environment                       0.271       0.090


polkadot/node/subsystem-bench/examples/availability_read.yaml #3

Network usage, KiB                     total   per block
Received from peers               424379.000  141459.667
Sent to peers                        703.000     234.333

CPU usage, s                           total   per block
availability-recovery                 51.128      17.043
Test environment                       0.502       0.167

$ cargo run -p polkadot-subsystem-bench --release -- --ci test-sequence --path polkadot/node/subsystem-bench/examples/availability_read.yaml | tee output.txt
$ cat output.txt
- benchmark_name: 'polkadot/node/subsystem-bench/examples/availability_read.yaml #1'
  network:
  - resource: Received from peers
    total: 509011.0
    per_block: 169670.33333333334
  - resource: Sent to peers
    total: 220.0
    per_block: 73.33333333333333
  cpu:
  - resource: availability-recovery
    total: 31.845848445
    per_block: 10.615282815
  - resource: Test environment
    total: 0.23582828799999941
    per_block: 0.07860942933333313

- benchmark_name: 'polkadot/node/subsystem-bench/examples/availability_read.yaml #2'
  network:
  - resource: Received from peers
    total: 411738.0
    per_block: 137246.0
  - resource: Sent to peers
    total: 351.0
    per_block: 117.0
  cpu:
  - resource: availability-recovery
    total: 18.93596025099999
    per_block: 6.31198675033333
  - resource: Test environment
    total: 0.2541994199999979
    per_block: 0.0847331399999993

- benchmark_name: 'polkadot/node/subsystem-bench/examples/availability_read.yaml #3'
  network:
  - resource: Received from peers
    total: 424548.0
    per_block: 141516.0
  - resource: Sent to peers
    total: 703.0
    per_block: 234.33333333333334
  cpu:
  - resource: availability-recovery
    total: 16.54178526900001
    per_block: 5.513928423000003
  - resource: Test environment
    total: 0.43960946299999537
    per_block: 0.14653648766666513

@AndreiEres AndreiEres added R0-silent Changes should not be mentioned in any release notes T12-benchmarks This PR/Issue is related to benchmarking and weights. labels Jan 31, 2024
@AndreiEres AndreiEres changed the title subsystem-bench: Collect output to a struct subsystem-bench: Prepare CI output Feb 1, 2024

if let Some(agent_running) = agent_running {
let agent_ready = agent_running.stop()?;
agent_ready.shutdown();
}

let output = if self.ci { serde_yaml::to_string(&vec![usage])? } else { usage.to_string() };
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: This logic is duplicated you can move it in a small funciton.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right. But we're going to remove the last branch and keep only run with test files. So this duplication is going to be removed as well.

@@ -99,6 +99,10 @@ struct BenchCli {
/// Enable Cache Misses Profiling with Valgrind. Linux only, Valgrind must be in the PATH
pub cache_misses: bool,

#[clap(long, default_value_t = false)]
/// Shows the output in YAML format
pub ci: bool,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably we can find a better name here like:

--output-format=yaml
--output-format=json

or even simpler

--yaml-output

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can start with more obvious --yaml-output, because I have no plans for additional json one :-)

Copy link
Contributor

@sandreim sandreim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! I also suggest to include the test objective in the output.

polkadot/node/subsystem-bench/src/core/environment.rs Outdated Show resolved Hide resolved
@AndreiEres
Copy link
Contributor Author

I also suggest to include the test objective in the output.

I think we add it as a title here, or you mean something else?

pub async fn benchmark_availability_read(
	benchmark_name: &str,
	env: &mut TestEnvironment,
	mut state: TestState,
) -> BenchmarkUsage {

@sandreim
Copy link
Contributor

sandreim commented Feb 6, 2024

I also suggest to include the test objective in the output.

I think we add it as a title here, or you mean something else?

pub async fn benchmark_availability_read(
	benchmark_name: &str,
	env: &mut TestEnvironment,
	mut state: TestState,
) -> BenchmarkUsage {

Yes, currently you print just the seq file name.

@AndreiEres AndreiEres added this pull request to the merge queue Feb 6, 2024
Merged via the queue into master with commit 9e6298e Feb 6, 2024
125 checks passed
@AndreiEres AndreiEres deleted the AndreiEres/subsystem-bench-output branch February 6, 2024 17:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
R0-silent Changes should not be mentioned in any release notes T12-benchmarks This PR/Issue is related to benchmarking and weights.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants