[Meta] Catch's performance

[horenmar]

As well know, Catch's performance, at both runtime and compile time, could be better.

Some changes are on their way to master, but more could be always done. To that end, benchmarks (self-contained test suites) that show pathological usage patterns, representative usage patterns or just unexpected slowness, are welcome.

Obviously, so are PR's against dev-performance branch.

[horenmar]

There is a request for special macros that only check for equality, without having expression decomposition, etc in #668.

[horenmar]

After merging in dev-performance-round2, I did some benchmarking, comparing current master to last release (1.6.1).

Using 3 benchmarks

1. A long loop (2^20 iterations) with extremely simple REQUIRE condition.
2. A long loop (2^20 iterations) with simple CHECK condition, that fails.
3. Full benchmark suite of the Json for Modern C++ project (version 2.1.0, contains ~11M assertions).

I ended with these numbers

Benchmark	1.6.1	master
Passing tests	3.4s	0.7s
Failing tests	15.5s	15.1s
JSON for modern C++	6m 25s	4m 14s

I also ran compile time benchmarks for the JSON tests and there has definitely been a regression in compile time. Compiling all the tests from nothing with Catch 1.6.1 takes ~15minutes, 25 seconds. Compiling with current master takes ~16 minutes.

[philsquared]

Not bad! And nice use of a real world project

[GatoRat]

FYI, the complaint I was given by one build engineer last year was that the compile time was becoming too long. I suggested precompiled headers, but he wasn't sure how to do it right on Linux.

[horenmar]

@GatoRat Yeah, there definitely has been some compile time regression (I added compile time information to the post above), as the framework gains more features there is more code that the compiler has to chug through. If you have some specific code that has gotten significantly worse, please share.

Otherwise, the compile times should get a bit better soon, Phil has some simplification in store.

[horenmar]

The new compile time flag (CATCH_CONFIG_FAST_COMPILE) speeds up test compilation on linux by ~10%. Windows benchmark pending.

[horenmar]

Another ~15% speedup for test compilation using CATCH_CONFIG_FAST_COMPILE was achieved in 7a8a020.