Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow choice of reference benchmark #577

Closed
christianhujer opened this issue Oct 8, 2022 · 5 comments
Closed

Allow choice of reference benchmark #577

christianhujer opened this issue Oct 8, 2022 · 5 comments

Comments

@christianhujer
Copy link

christianhujer commented Oct 8, 2022

When running hyperfine, we get a nice output.

For example, when running hyperfine like this:

hyperfine -L shell sh,ash,ksh,bash,zsh,csh,fish,tcsh,nu,pwsh '{shell} -c "echo \"Hello, world!\""'

The Summary of the output looks like this:

Summary
  'sh -c "echo \"Hello, world!\""' ran
    1.23 ± 0.73 times faster than 'ash -c "echo \"Hello, world!\""'
    1.90 ± 0.45 times faster than 'ksh -c "echo \"Hello, world!\""'
    2.03 ± 1.18 times faster than 'bash -c "echo \"Hello, world!\""'
    2.41 ± 0.55 times faster than 'zsh -c "echo \"Hello, world!\""'
    6.48 ± 1.29 times faster than 'csh -c "echo \"Hello, world!\""'
    7.26 ± 1.62 times faster than 'fish -c "echo \"Hello, world!\""'
    7.93 ± 1.68 times faster than 'tcsh -c "echo \"Hello, world!\""'
   26.55 ± 5.52 times faster than 'nu -c "echo \"Hello, world!\""'
  674.56 ± 131.91 times faster than 'pwsh -c "echo \"Hello, world!\""'

It would be really great if it were possible to pick one of the benchmark runs as "reference". For example, if I'm interested in how nu performs in comparison, I would like it to be on top, and the other benchmarks should be listed as times faster or times slower, depending on their performance.

In this example, when I run it like this:

hyperfine -L shell sh,ash,ksh,bash,zsh,csh,fish,tcsh,nu,pwsh --reference nu '{shell} -c "echo \"Hello, world!\""'

Then the Summary should look like this:
Summary

  'nu -c "echo \"Hello, world!\""' ran
    26.55 ± 5.52 times slower than 'sh -c "echo \"Hello, world!\""'
    21.58 ± 7.56 times slower than 'ash -c "echo \"Hello, world!\""'
    13.97 ± 12.27 times slower than 'ksh -c "echo \"Hello, world!\""'
    13.08 ± 4.68 times slower than 'bash -c "echo \"Hello, world!\""'
    11.02 ± 10.04 times slower than 'zsh -c "echo \"Hello, world!\""'
    4.10 ± 4.27 times slower than 'csh -c "echo \"Hello, world!\""'
    3.66 ± 3.40 times slower than 'fish -c "echo \"Hello, world!\""'
    3.35 ± 3.29 times slower than 'tcsh -c "echo \"Hello, world!\""'
   25.41 ± 23.90 times faster than 'pwsh -c "echo \"Hello, world!\""'
@christianhujer christianhujer changed the title Feature Request: pick a reference Feature Request: Allow choice of reference benchmark Oct 9, 2022
@sharkdp
Copy link
Owner

sharkdp commented Oct 13, 2022

That sounds like a very useful feature request, thank you.

How would this work in cases where we do not have a -L option, but rather multiple commands? would the reference be a number? or the full command?
It would be great if we could come up with a proper proposal for the complete command-line interface. And evaluate how this might interfere/combine with other command line options of hyperfine.

@Animeshz
Copy link

Animeshz commented Oct 15, 2022

I believe --reference should always contain a complete command (and separate from normal run parameters, notice no nu in -L shell):

hyperfine --reference 'nu -c "echo \"Hello, world!\"' -L shell sh,ash,ksh,bash,zsh,csh,fish,tcsh,pwsh '{shell} -c "echo \"Hello, world!\"'

This way we can also measure it against not necessarily same programs.

@sharkdp
Copy link
Owner

sharkdp commented Oct 29, 2022

@christianhujer what do you think?

@Julian
Copy link

Julian commented Jan 17, 2023

I'm finding this issue while searching for a related one -- going to mention that here since it may be relevant to designing --reference, though of course the addition may be deemed out of scope:

But in my case, I want a reference command per parameter I think. Imagine the use case being if I were similarly testing 10 shells running some command, and I want to measure how slowly running the command is relative to a reference point, where the reference point is each shell's own startup time, and therefore basically normalize each shell by how long it took to start up, thereby measuring just the command execution.

(Of course I'd separately want to measure them against each other in absolute value, but sometimes I care just about the runtime once the shell is up. And --setup for what it's worth isn't really appropriate either in my real case, I want to indeed measure the two commands).

But yeah -- is there any thoughts on whether letting --reference itself include the parameter from -L?

@sharkdp sharkdp changed the title Feature Request: Allow choice of reference benchmark Allow choice of reference benchmark Apr 17, 2023
@sharkdp
Copy link
Owner

sharkdp commented Jun 30, 2024

Implemented by @JanEricNitschke in #744

@sharkdp sharkdp closed this as completed Jun 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants