Bug: wildly fluctuating test times

[sandeep-datta]

Edit: I just realized I may have been looking at the wrong metric. What does the 2 us in Test PASSED (10001/10001) [ 2 us] stand for?

General issue
Classification: Buggy performance testing code.
Reproducible test case:
./repro # https://gist.github.com/sandeep-datta/f9706238e7a65c0d6aad545fde533aae
Filtered output: https://gist.github.com/sandeep-datta/e3f956fda7210a401fbc70ab3dddd0b3
Filter applied to saved output: grep Test output | grep us | sort -k 2

Suggested fix: run the test 100 times (for any test that takes less than 1 ms to run), remove outliers and show the average of collected run times.

[4x3lpri3t0]

I believe that 2 us stands for 2 microseconds (0.002 miliseconds).

[metopa]

Hi @sandeep-datta , thank you for noticing that. Right now we're in the process of rewriting internal infrastructure. After that we'll improve the precision of measuring execution time.