Show output only on fails

[arcanis]

I generally don't care about the output, except when the process aborts. When that happens, I want to understand the reason of the exit code. Just adding --show-output would waste resources and make debug a tad harder.

[sharkdp]

Interesting request - thank you. Implementing this, however, would force us to always collect the output (just in case the process could fail). I'm not sure if we want to do that by default (related discussion: #377), as it could lead to a slowdown of the benchmarked process, compared to what we do right now (pipe to /dev/null).

[mralusw]

A small improvement would be to have -r 1 simply run the commands with --show-output implied (no benchmark). It makes it easier to hit <up>, change a few characters in the command line, and hit <cr> — which is how most of us debug errors. Currently -r 1 just refuses to run.

As food for thought, -r 0 and -r -N (negative) could also be given meaningful semantics (I'm not sure what semantics though)

[avar]

@arcanis @sharkdp @mralusw : Can't this proposed feature be generalized to a shell alias that does this for you:

$ exec 4>&2 3>&1 ; hyperfine -i -s basic -r 1 -L n 0,1 'PATH=/bin:/usr/bin /usr/bin/chronic sh -c "echo about to die with code {n}; bad-cmd-{n}; exit {n}" >&3 2>&4'
Benchmark 1: PATH=/bin:/usr/bin /usr/bin/chronic sh -c "echo about to die with code 0; bad-cmd-0; exit 0" >&3 2>&4
  Time (abs ≡):         26.5 ms               [User: 26.8 ms, System: 0.0 ms]
 
Benchmark 2: PATH=/bin:/usr/bin /usr/bin/chronic sh -c "echo about to die with code 1; bad-cmd-1; exit 1" >&3 2>&4
about to die with code 1
/bin/sh: 1: bad-cmd-1: not found
  Time (abs ≡):         26.4 ms               [User: 26.7 ms, System: 0.0 ms]
 
  Warning: Ignoring non-zero exit code.
 
Summary
  'PATH=/bin:/usr/bin /usr/bin/chronic sh -c "echo about to die with code 1; bad-cmd-1; exit 1" >&3 2>&4' ran
    1.00 ± 0.00 times faster than 'PATH=/bin:/usr/bin /usr/bin/chronic sh -c "echo about to die with code 0; bad-cmd-0; exit 0" >&3 2>&4'

On Debian you'll need apt get moreutils first.

It's a good use-case, but you're generally better off with using standard shell features when possible. You'll get funny interleaving of the output with that naïve trick, which can be (I think) hacked around with either bash/zsh-specific syntax, or clever use of mkfifo.

[sharkdp]

--runs=1 is now supported via #447. It does not imply --show-output, though.

[sharkdp]

It seems we cannot capture stdout by default without performance hits, see #377. Assuming this can not be fixed somehow, I'd like to close this ticket.