faster multiproject test runs with parallelization

[keturn]

One way to spend less time waiting for tests is to throw more hardware at them.

#4417 adds parallel test execution to the tests for engine. Workspaces where you have a lot of modules checked out at once (i.e. omega) could do a similar thing at a higher level, running compilation and testing tasks in multiple modules in parallel.

This involves scheduling execution across multiple subprojects, so it'll be done at the level of Gradle configuration rather than Jupiter.

We have a lot of modules. We would probably get more than enough parallelization by gradle parallel task execution without needing to configure anything specific to testing: https://docs.gradle.org/6.8.3/userguide/performance.html#parallel_execution

If the bottleneck is one module with a lot of slow tests, gradle also offers has a parallelization setting specifically for tests: https://docs.gradle.org/current/userguide/java_testing.html#sec:test_execution

Success metrics:

• Is the result actually faster?
  • it might not be! maybe the code under test already uses enough threads that we don't get more CPU utilization out of parallelizing them. Maybe the bottleneck is something else.
  • how much faster for different size workspaces?
  • how many CPU cores do you need for parallelization to be worth it?
• Does it cause reliability problems?
  • we don't have to worry about in-memory conflicts because they will be in different processes. Issues with the filesystem or network ports are still possible.

[keturn]

I tried running tests in omega using the --parallel command line flag here on my i7-6700 workstation (4 physical cores + hyperthreading).

gradlew clean :extractConfig :extractNatives test --parallel --scan

Key takeaways:

• ItemPipes tests sometimes stall, parallel or otherwise.
• Yes, it is faster. When things go well it's under 10 minutes (including compilation!) vs 16 or more.
• Yes, it does cause problems sometimes. There is an error I haven't seen from other test runs: Connection refused: localhost/127.0.0.1:25777. Probably conflicts between concurrent instances of MTE.
• The thing that takes the most time by far is MTE's own test suite. Running module tests in parallel is not worth it if it means you have to re-try the MTE suite, because re-running the most expensive piece wipes out any time you'd saved.
• 16 minutes versus 10 minutes is nice, but it's not a quantum leap.

Other notes:

• loadavg went over 16. gradle is over-optimistic about how many of these it can do in parallel. Can we adjust its expectations? Or is it self-adjusting as it gets more experience with the build?
• if a test stalls, don't Ctrl-C the terminal, that can shut the whole run down. Instead try to find the specific java process that's stalling and SIGTERM that.
• Your mileage may vary with other hardware. Seems unlikely to be worth it with fewer than 4 physical cores.

[keturn]

In the interest of having individual actionable tickets for people to pick up, follow-up issues have been made:

• "Connection refused" errors when attempting parallel test execution Terasology/ModuleTestingEnvironment#48
• MTE's own test suite is very slow Terasology/ModuleTestingEnvironment#49
• ItemPipes tests sometimes stall Terasology/ItemPipes#37

[jdrueckert]

parallelize module unit tests