Opt-in support for parallel execution

[marcphilipp]

Related to #90.

[ptahchiev]

+1 for this one. TestNG has had this feature for a long time and the maven surefire plugin supports parallel tests out of the box:

http://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-options-and-parallel-execution.html

[radai-rosenblatt]

+1. would definitely help in shortening build times

[leonard84]

Will this be part of the platform, or the engine, or both?

While thinking about better support for parallel execution in Spock, I identified some problems that should be handled by the testing framework.

• Parallel (integration) testing in conjunction with global state (system properties/db/...) can leak into other tests
  • e.g., Spock has the @RestoreSystemProperties extension, which will save the SystemProperties before the execution and restore it after the test. This works fine for single threaded execution, but in concurrent test execution this will lead to race conditions.
  • The bad solution would be to disable concurrent execution

My idea is that each test (class/method) could declare its use of global resources, e.g. @UseResource(value="SystemProperties", mode=ReadWrite). The engine would keep a list of ReentrantReadWriteLock for each of the resources and acquire/release those locks accordingly. Care would have to be taken to avoid deadlocks, by ordering the lock acquisition accordingly.

This way the framework could offer the maximum concurrency while still offering providing tools to isolate problematic tests.

There would be a list of predefined resources, such as Global (all tests would implicitly acquire a ReadLock for this) or SystemProperties, but users can define their own to match their application.

Thoughts? Of course all names are open for change.

[marcphilipp]

I think an engine needs to decide on which level tests can be executed in parallel. However, there are quite a few similarities between different testing frameworks where the platform could provide support, e.g. via a base implementation that multiple test engines can use.

I really like your reentrant lock idea! Is there already a TestEngine implementation for Spock? If so, do you use the HierarchicalTestEngine abstraction? I wrote a ForkJoinPool based prototype for it some time ago.

[leonard84]

@marcphilipp no Spock doesn't have it's own TestEngine yet. As better support for parallelism would be a major part in it, I'd wanted to check in on this first before running in the wrong direction/doing redundant work. Switching to a own TestEngine will be done with Spock 2.0

[wlsc]

Any news on that topic?

[sormuras]

[...] And is it possible execute it with maven-surefire-plugin?

See the Maven instructions shown at jitpack URL above. If it works with Surefire? No idea. Keen on trying it out?

[sormuras]

@marcphilipp The https://github.com/junit-team/junit5/tree/experiments/parallel-execution branch is still based on 5.1-M2 ... it lacks the final org.junit.jupiter.api.condition package and other fixes. Will @leonard84 or you rebase it to master in the near future?

[marcphilipp]

I'll meet with @leonard84 this weekend. I'll try to merge master into the feature branch before.

[marcphilipp]

I've just updated the feature branch. Feel free to try it out using Jitpack.

[dlanaghen]

Parallel testing is a big deal to us. I have experimented with your version and my first simple tests look real promising. Is there a ball park for when this might get merged into Junit5 and when an official release timeframe might be? If this is going to be real, I might forge ahead and begin really exercising it.

[marcphilipp]

@dlanaghen Thanks for experimenting with it and letting us know!

We plan to include it as an experimental feature in 5.3 M1 which is currently scheduled for June 24. If no regressions or major issues are found, you can expect a RC and the GA release within two weeks from that date.

[dlanaghen]

@marcphilipp this is great news. With better parameterized testing features and your new multi-threaded test runner, you will have filled a couple of big holes in JUnit. Thanks for your work.

[koretzki]

@marcphilipp, great to hear, this (no parallelism) was a road blocker for us to adopt fully JUnit 5.

[markthegrea]

Where is the documentation on how to use this?

[sormuras]

@markthegrea It is still on the snapshot version of the User-Guide, because it is integrated in version 5.3-RC1 of the Jupiter Engine: https://junit.org/junit5/docs/snapshot/user-guide/#writing-tests-parallel-execution

When 5.3 is released, the current User-Guide will show it, too.