Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Starlark: Benchmarks --iterations <count> command line flag #12511

Closed
wants to merge 1 commit into from

Conversation

stepancheg
Copy link
Contributor

Option to run a fixed number of iterations to use benchmarks with
an external comparison tool.

This is a follow-up to
this comment.

Benchmarks --iterations 10000 will run 10000 iterations of a given
benchmark (or of the benchmarks if filter is not specified).

For example:

$ absh \
    -a "java -jar $HOME/Benchmarks_deploy-a.jar --filter bench_sort_small --iterations 10000000" \
    -b "java -jar $HOME/Benchmarks_deploy-b.jar --filter bench_sort_small --iterations 10000000"
...
A: N=53 r=3.992+-0.425 se=0.058
B: N=53 r=3.919+-0.384 se=0.053
B/A: 0.982

Note that currently Benchmarks parses all benchmark files (creating
unnecessary overhead), this will be fixed when new
--filter <file>:<benchmark> syntax is introduced.

@google-cla google-cla bot added the cla: yes label Nov 18, 2020
@stepancheg stepancheg force-pushed the bench-iter branch 2 times, most recently from 6011aaa to 0c69e2b Compare November 19, 2020 01:04
@stepancheg stepancheg force-pushed the bench-iter branch 2 times, most recently from b593987 to 0faf29b Compare November 26, 2020 16:45
@stepancheg
Copy link
Contributor Author

Review.

formatDuration(((double) b.cpu) / b.count),
b.steps / b.count,
formatBytes(b.alloc / b.count));
b.count != 0 ? formatDuration(((double) b.time) / b.count) : "?",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I should have foreseen that the zero case would be problematic. Let's reject iterations=0, but continue to use -1 as the sentinel value meaning "unset".

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ping... once this is addressed I'll import the PR.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Option to run a fixed number of iterations to use benchmarks with
an external comparison tool.

This is a follow-up to
[this comment](bazelbuild#12498 (comment)).

`Benchmarks --iterations 10000` will run 10000 iterations of a given
benchmark (or of the benchmarks if filter is not specified).

For example:

```
$ absh \
    -a "java -jar $HOME/Benchmarks_deploy-a.jar --filter bench_sort_small --iterations 10000000" \
    -b "java -jar $HOME/Benchmarks_deploy-b.jar --filter bench_sort_small --iterations 10000000"
...
A: N=53 r=3.992+-0.425 se=0.058
B: N=53 r=3.919+-0.384 se=0.053
B/A: 0.982
```

Note that currently `Benchmarks` parses all benchmark files (creating
unnecessary overhead), this will be fixed when new
`--filter <file>:<benchmark>` syntax is introduced.
@bazel-io bazel-io closed this in 92a9e2f Dec 7, 2020
@stepancheg stepancheg deleted the bench-iter branch December 7, 2020 18:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants