-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Starlark: Benchmarks --iterations <count> command line flag #12511
Conversation
6011aaa
to
0c69e2b
Compare
b593987
to
0faf29b
Compare
Review. |
formatDuration(((double) b.cpu) / b.count), | ||
b.steps / b.count, | ||
formatBytes(b.alloc / b.count)); | ||
b.count != 0 ? formatDuration(((double) b.time) / b.count) : "?", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry, I should have foreseen that the zero case would be problematic. Let's reject iterations=0, but continue to use -1 as the sentinel value meaning "unset".
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ping... once this is addressed I'll import the PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
Option to run a fixed number of iterations to use benchmarks with an external comparison tool. This is a follow-up to [this comment](bazelbuild#12498 (comment)). `Benchmarks --iterations 10000` will run 10000 iterations of a given benchmark (or of the benchmarks if filter is not specified). For example: ``` $ absh \ -a "java -jar $HOME/Benchmarks_deploy-a.jar --filter bench_sort_small --iterations 10000000" \ -b "java -jar $HOME/Benchmarks_deploy-b.jar --filter bench_sort_small --iterations 10000000" ... A: N=53 r=3.992+-0.425 se=0.058 B: N=53 r=3.919+-0.384 se=0.053 B/A: 0.982 ``` Note that currently `Benchmarks` parses all benchmark files (creating unnecessary overhead), this will be fixed when new `--filter <file>:<benchmark>` syntax is introduced.
0faf29b
to
08a072c
Compare
Option to run a fixed number of iterations to use benchmarks with
an external comparison tool.
This is a follow-up to
this comment.
Benchmarks --iterations 10000
will run 10000 iterations of a givenbenchmark (or of the benchmarks if filter is not specified).
For example:
Note that currently
Benchmarks
parses all benchmark files (creatingunnecessary overhead), this will be fixed when new
--filter <file>:<benchmark>
syntax is introduced.