Introduce support for parallel test execution in declarative test suites

[casid]

I'm currently migrating a project from JUnit4 to JUnit5 (Branch: https://github.com/casid/mazebert-ladder/tree/junit5).

A feature of the JUnit4 tests is that I can run all unit tests in parallel. This is an essential feature since it a) chops down the test execution time significantly and b) prevents developers from adding shared state to unit tests.

In JUnit4 with the help of junit-toolbox:

@RunWith(ParallelSuite.class)
@SuiteClasses("**/*Test.class")
public class UnitTestSuite {
}

I wished JUnit5 to support something like this:

@Suite(parallel = true)
@SuiteClasses("**/*Test")
public class UnitTestSuite {
}

But I couldn't find anything that looked even slightly like it.

I think suites could greatly improve configuration options for test executions. In fact, it would remove the need for external tools to implement the same features (e.g. parallel execution / test discovery). You could simply point maven-surefire, gradle or IntelliJ to the TestSuite(s) you want to execute.

Integration tests could benefit from this too:

@Suite(parallel = true)
@SuiteClasses("**/*IT")
@IncludeTags("parallel")
public class ParallelIntegrationTestSuite {
    @BeforeClass
    public static void setUp() throws Exception {
        TestDataSourceProvider.instance.prepare();
    }

    @AfterClass
    public static void tearDown() throws Exception {
        TestDataSourceProvider.instance.dispose();
    }
}

I would love to have something like this in JUnit5! Currently I have some mixed feelings how to proceed with adopting JUnit5 (great new features like nested tests) vs JUnit4 (great test execution control).

Related Issues

• Introduce @BeforeSuite and @AfterSuite annotations #456
• Introduce an extension point to allow test container creation #687
• Introduce support for declarative test suites #744
• Improve JavaDoc for junit-platform-suite-api annotations #746

[sormuras]

Related to #60

[casid]

I don't think that this duplicates #60. There's not much information on how to achieve parallel execution, except a reference to #90 - which I think suggests to delegate this to external tools.

Instead, the idea here is to let JUnit handle it. For instance with suites.

[sormuras]

Understood. #60 still serves as a good umbrella issue for parallel, other thread, multi-thread feature requests.

[sbrannen]

This is closely related to #744.

[sbrannen]

I have reworded this issue's title accordingly.

[sbrannen]

Added Related Issues section.

[casid]

Thanks for looking into this. What's the status of #744? If things turn out to become more concrete, I would happily support the development.

[sbrannen]

@casid, #744 is currently in the 5.1 backlog.

So the team won't begin working on it until after 5.0 GA has been released.

[sbrannen]

If things turn out to become more concrete, I would happily support the development.

Well, you could of course start experimenting in your own branch, develop a proof of concept, etc.

[koretzki]

not sure if is part of the same issue,

I am trying to migrating a maven project from JUnit4 to JUnit5.
I am using maven-surefire-plugin and maven-failsafe-plugin to run tests.

When I added the needed dependencies in pom.xml file for the plugins,
the tests do run but they do not execute in parallel like they used to do without the dependencies (the plugins have the thread use in their configuration).
Without the dependencies maven won't pick up and run Junit jupiter tests but do run Junit Vintage tests in parallel.

Is there any support for parallel test executions in maven ?
if not, is it part of this issue or should there be a new issue ?

[marcphilipp]

Regarding Maven: parallel is not supported at the moment, but forkCount should work.

@koretzki Can you please try that?

[koretzki]

Yes, it does work.
But I require it to run as threads, runing in forks might cause issue with our environment.
Do you know when it would be supported in Maven or Gradle ?

[marcphilipp]

The plan is to handover the plugin to the Maven Surefire team. Please consult them regarding their roadmap. /cc @Tibor17

[JLLeitschuh]

Is this currently supported in the gradle plugin? Is the team waiting for the Gradle team to take over the Junit5 gradle integration and embed it into the gradle tool by default.

[marcphilipp]

Our Gradle plugin does currently not support parallel test execution. And, indeed, we're waiting for the Gradle team to start work on first-class support for JUnit 5. As a sidenote, there's also a prototype for parallel execution within the Jupiter engine, see #60.

[jamiewastin]

@marcphilipp - Now that there is gradle plugin support via the junit-platform-gradle-plugin does this pave the way for parallel test support?

My specific use-case is to be able to run each test in its own isolated JVM to ensure that static state is not carried between unit tests (legacy system). This seems closely related to parallel test support.

[JLLeitschuh]

@jamiewastin
Consider using the new Junit 5 support that is now built into Gradle 4.6.
When configuring your test you can set the maxParallelForks on the Test task.
https://docs.gradle.org/current/javadoc/org/gradle/api/tasks/testing/Test.html#setMaxParallelForks-int-

[casid]

The parallel test execution feature is exactly what I needed. This is an amazing and truly elegant solution!

For my use case this worked perfectly:

Create junit-platform.properties in src/test/resources:

junit.jupiter.execution.parallel.enabled = true
junit.jupiter.execution.parallel.mode.default = concurrent
junit.jupiter.execution.parallel.mode.classes.default = concurrent

Add @Execution(ExecutionMode.SAME_THREAD) to classes where methods cannot be run in parallel.

The concurrent execution of test methods (not just classes) is amazing.

Thank you so much for adding this feature! I'm closing this issue now.

[sbrannen]

@casid, thanks for letting us know.

We are very glad to hear your excitement about the parallel test execution support.

[koretzki]

does it work with recent version of Maven Surefire/ Failsafe plugins ?
based on Maven docs here
https://maven.apache.org/surefire/maven-failsafe-plugin/examples/junit-platform.html
the presence of JUnit 5 tests triggers a test runner called "JUnit 5 Platform Engine"
which does not support running tests in parallel.
I tried to with the issue a number of times, even forcing a runner Junit47 to run.
could not make it to run parallel with Junit 5.
Junit 5.3-5.4 had the junit-platform-runner it did work. but alerts on depreciation on Maven, 2.22.0+ .
Any solutions to Parallel execution in coming Maven version / Junit 5.5 ?

[sormuras]

@koretzki It does work with Maven Surefire/Failsafe.

Consult the User Guide how to a) enable parallel execution in Jupiter and b) how to configure Maven Surefire.

The Maven documentation states "Running Tests in Parallel -- From JUnit Platform does not support running tests in parallel." due to the fact that the <parallel>+*</parallel> has no effect. The JUnit Platform doesn't provide such a feature.

[Tibor17]

@koretzki
If you want to run the JUnit5 in Maven Surefire you have to add engine in dependencies, see https://maven.apache.org/surefire/maven-surefire-plugin/examples/junit-platform.html
In the next coming version 3.0.0-M4 adding the jupiter-api would be enough.

You can use the latest version 3.0.0-M3 of Surefire with parallel execution and the tests will be executed but the statistics in test result will be very messy as well as the log. We would like to fix it or rework some Surefire code in order to fix it in 3.0.0-M4.

[koretzki]

Thanks @Tibor17 and @sormuras this helped my with surefire plugin to work.
I tried the same with the Maven Failsafe plug-in but it didn't work,
It ran but not in parallel.
Should it work in Failsafe, or it not supported yet ?

[Tibor17]

@koretzki The real execution should be parallel. The only problem is that sometime the reports are lost or incomplete. Is this happening?

[koretzki]

Hi @Tibor17, reports are ok, but Failsafe does not run in parallel. I took my parallel working surefire maven profile and replaced just it with the Failsafe plugin. Any printout that was with thread name changed from ForkJoinPool (names) to Main in all test printouts (same check with thread ids).
tried failsafe versions 2.22.2 to 3.0.0-M4.
Junit 5.5.2 and 5.6.0-RC1.
pom file configuration params :

<configurationParameters>
    junit.jupiter.execution.parallel.enabled = true
    junit.jupiter.execution.parallel.config.dynamic.factor = 1 
    junit.jupiter.execution.parallel.mode.default = concurrent
    junit.jupiter.execution.parallel.mode.classes.default =concurrent
</configurationParameters> 

[marcphilipp]

@koretzki Does it work if you put those properties into src/test/resources/junit-platform.properties?

[koretzki]

@marcphilippI I did try it, but it did not work.

[GethDeeo]

@marcphilipp My bad to chime in, but this issue more or less describes my problem, and I saw that the last mention of you contains a typo :)

So I'm trying to reproduce a working parallel suites scenario from JUnit 4 to JUnit 5.
Ok, so I know that Surefire <parallel> is off the table.
In itself, parallel execution is working fine in JUnit 5.
In itself, suites are working fine in JUnit 5.
I see that junit-platform-suite implements TestEngine instead of HierarchicalTestEngine, and only the latter contains code for parallelism.

Are there any plans to change this? I have a large testbase so I desperately need parallelism, but within a "unit" (suite) a single threaded, fixed-order execution of test classes is a must. These are NOT unit tests, but functional tests with webdriver - the test classes/methods can be executed independently, but without shared states (same thread) and predetermined order, execution takes a very, very long time.
(I've somewhat succeeded with Spock by dynamically adding resource locks with package names (so package == suite), but Spock does not guarantee class order at all - and for optimal execution time, also the "suites" should be ordered.)