Benchee Warnings

Fast execution warning

This warning occurs when an individual invocation of the function to benchmark takes less than 10μs (usually, it's 10 "whatever accuracy your operating system can measure in). This is warning worthy as measurements occur in microseconds and in integers so measuring it would be rather inaccurate.

When this happens Benchee won't measure individual runs but rather run it a couple of times and report the average back. Measures will still be correct, but the overhead of running it n times goes into the measurement. Also statistical results aren't as good, as they are based on averages now. You can see an example of too fast functions in this sample.

Especially memory measurements may be way off due to the overhead introduced be repeating the function call etc.

When running with hooks it is even worse as we have to now measure the hooks as part of the function invocation

To remedy the situation, if possible, increase the input size so that an individual run takes more time and the warning disappears.

Also please note that this isn't a limitation specific to Benchee - all benchmarking tools are affected by it as far as I know and if someone comes up with a counter measure then I'm eager to implement it myself :)