Feature Request: Matrix measurements #253
Comments
|
Thank you very much for your feedback. I agree that this would be a useful feature to have. Implementation-wise, we could probably take a first step by converting the "matrix" to a flat list. This would mean that we would not add special support in the terminal output, the export formats, etc. It would simply be a list of benchmarks. 27 in your example: |
|
Expanding a bit on this, I think there would be at least two modes of lists, zip (i.e. pair up the variable parameters) and cross product (all combinations of values). It's not a given that you would want to support both I guess? EDIT: I have previously developed a similar and much worse tool than hyperfine for benchmarking, which had significantly more complicated options for combining inputs. I would possibly be interested in working on this for hyperfine, depending on some factors mostly boiling down to how much time it would take to do all-right, if anyone else is available to work on it, and how much time I have. |
|
Thank you for the feedback. I would go with the cross-product version only, I think. I think the zipped-version is much less common and (while not pretty), it probably can be simulated in most cases: hyperfine \
--parameter-list args "--foo 1 --bar a","--foo 2 --bar b"
"my-command {args}" |
|
Also interested by this feature, in my case, I was expecting a cross product: Zipping would also be nice, but could be added separately with a new syntax, e.g.: |
|
I have been thinking about this. My own use case is to race N implementations of SMT solvers against each other for sets of benchmarks (input files). If N=2 and I am A/B testing (which would be my typical use case during development), this would in principle be equivalent to a shell for loop executing a regular "compare two" input to Hyperfine with the benchmark files coming from the loop. However, the situation becomes a bit more interesting if I were to compare three or more implementations; then I would be interested in a comparison between the best and the second-best. The main design challenge I see to getting something useful (if not feature-complete) out the door is to let the user somehow control how the results are generated. For example, if I have parameters (algoA,algoB), (fileA,fileB), (optionA, optionB) I would like to predictably have an output like: I would consider this issue solved for my own use if I could just implement the for loop solution directly in Hyperfine. What are your thoughts, @sharkdp? |
This patch permits the `-L`/`--parameter-list` argument to appear multiple times. Each additional parameter increases the dimensionality of the parameter matrix. One benchmark will be run for each combination of parameters (i.e., the benchmark space is the Cartesian product of the parameter spaces). For now, `--parameter-list` and `--parameter-scan` are still mutually exclusive. If desired, a follow-up change could similarly permit multiple occurrences of `--parameter-scan` and also permit use of both listed and scanned parameters together. (After all, `--parameter-scan` can be thought of as syntactic sugar for a parameter list.) This implementation is a little profligate with memory usage for `BenchmarkResult`s: each result contains a separate copy of all parameter names (as `String`s). This could be done away with by threading lifetimes through the `BenchmarkResult`s, or by ref-counting the names. Because `BenchmarkResult`s now have maps with arbitrary key sets, we can no longer use `serde` to serialize them to CSV. But a CSV writer is easy to write by hand, so we just do that. Fixes sharkdp#253. Test Plan: Some unit tests included. As an integration test, try: ``` cargo run --release -- --runs 10 \ -L foo foo1,foo2 -L bar bar9,bar8 \ 'echo {foo} {bar}' \ 'printf "%s\n" {foo} {bar}' \ --export-csv /tmp/out ``` with all the export formats. wchargin-branch: param-list-matrix wchargin-source: 11051ab222c9de0fbc6ac6a080fbca48e05996c2
|
I’ve sent a pull request for this over at #318, which implements a cross |
This patch permits the `-L`/`--parameter-list` argument to appear multiple times. Each additional parameter increases the dimensionality of the parameter matrix. One benchmark will be run for each combination of parameters (i.e., the benchmark space is the Cartesian product of the parameter spaces). For now, `--parameter-list` and `--parameter-scan` are still mutually exclusive. If desired, a follow-up change could similarly permit multiple occurrences of `--parameter-scan` and also permit use of both listed and scanned parameters together. (After all, `--parameter-scan` can be thought of as syntactic sugar for a parameter list.) This implementation is a little profligate with memory usage for `BenchmarkResult`s: each result contains a separate copy of all parameter names (as `String`s). This could be done away with by threading lifetimes through the `BenchmarkResult`s, or by ref-counting the names. Because `BenchmarkResult`s now have maps with arbitrary key sets, we can no longer use `serde` to serialize them to CSV. But a CSV writer is easy to write by hand, so we just do that. Fixes #253. Test Plan: Some unit tests included. As an integration test, try: ``` cargo run --release -- --runs 10 \ -L foo foo1,foo2 -L bar bar9,bar8 \ 'echo {foo} {bar}' \ 'printf "%s\n" {foo} {bar}' \ --export-csv /tmp/out ``` with all the export formats. wchargin-branch: param-list-matrix wchargin-source: 11051ab222c9de0fbc6ac6a080fbca48e05996c2
|
Released in hyperfine v1.11.0. |
If I want to benchmark my program with one variable command argument, I can use
--parameter-list. On the other hand, if I want to benchmark four, five, six variables, it can get tedious.It would be nice if I'd be able to run:
The text was updated successfully, but these errors were encountered: