JUnit Jupiter @Nested tests do not run when selecting enclosing class with Surefire

[mwiercin]

Overview

Per Marc Philipp's request on Stack Overflow:

I'm trying to use JUnit Jupiter in my side project as a trial before migrating my main project. I'd like to use @Nested tests to make my test classes cleaner.

Everything is fine when I ran my test suite as a whole. However, as soon as I try running just a single test, @Nested ones are not executed.

mvn -Dtest=com.mycompany.test.MyTest surefire:test

Using JUnit 5.1.0, JUnit platform 1.1.0

<build>
<plugins>
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>2.19.1</version>
    <dependencies>
      <dependency>
        <groupId>org.junit.platform</groupId>
        <artifactId>junit-platform-surefire-provider</artifactId>
        <version>${org.junit.platform.version}</version>
      </dependency>
      <dependency>
        <groupId>org.junit.jupiter</groupId>
        <artifactId>junit-jupiter-engine</artifactId>
        <version>${org.junit.version}</version>
      </dependency>
    </dependencies>
  </plugin>
</plugins>
</build>

Test class:

public class MyTest {

  @Test
  public void thisTestExecutes() { }

  @Nested
  public class NestedTests {
    @Test
    public void thisTestDoesnt() { }
  }
}

Related Issues

• Maven Surefire Plugin does not execute static nested classes #1377
• Document how to execute static nested test classes with Maven Surefire #1378

[smoyer64]

This is an interesting use case - I don't believe it represents a bug in JUnit 5 or in Surefire - Surefire provides a generalized way to discover tests, filter them and then executes them individually. It adapts to each test engine via the Provider API.

When you run mvn test, Surefire delegates the discovery of the tests to JUnit 5, then filters them by using the complete class name (as you've provided) or by globbing each name for you. You should get the same behavior executing mvn -Dtest=MyTest surefire:test. Nested test classes are no different that other Java inner classes and so you end up with a class named MyTest$NestedTests on your classpath.

Surefire has recently added the ability to filter on method names to the test parameter, so you might expect you could run the inner classes tests with MyTest$NestedTests but when you do this, Surefire runs the outer class - it seems the filter specification doesn't provide a way to run tests in inner classes. This makes sense as I believe that JUnit 5 is the first Provider to support inner classes (I suspect tests in inner classes might run in JUnit 4 but they'd do so without the context of the outer class - note that I haven't tried this).

So, it appears that a feature request needs to be made to the Surefire project. This could go one of two ways:

1. Implicitly include inner classes contained by tests selected by the filter
2. Explicitly require users to list the inner classes they want to select but alter the filter to allow the user to supply inner class names.

[marcphilipp]

While Surefire provides the filter, it's the Provider's responsibility to apply it.

This is what we currently do:

junit5/junit-platform-surefire-provider/src/main/java/org/junit/platform/surefire/provider/TestMethodFilter.java

Lines 28 to 55 in 4c8c469

	class TestMethodFilter implements PostDiscoveryFilter {
	private final TestListResolver testListResolver;
	TestMethodFilter(TestListResolver testListResolver) {
	this.testListResolver = testListResolver;
	}
	@Override
	public FilterResult apply(TestDescriptor descriptor) {
	// @formatter:off
	boolean shouldRun = descriptor.getSource()
	.filter(MethodSource.class::isInstance)
	.map(MethodSource.class::cast)
	.map(this::shouldRun)
	.orElse(true);
	// @formatter:on
	return FilterResult.includedIf(shouldRun);
	}
	private boolean shouldRun(MethodSource source) {
	String testClass = TestListResolver.toClassFileName(source.getClassName());
	String testMethod = source.getMethodName();
	return this.testListResolver.shouldRun(testClass, testMethod);
	}
	}

I'm wondering whether we should add a special case when the test class is an inner class (i.e. it has an enclosing class and is not static), the TestListResolver does not have a method pattern and accepts any of the enclosing classes of the test class.

[smoyer64]

@marcphilipp I'd forgotten that the Provider actually does the filtering. Since Surefire is preprocessing the filters (at a minimum to do the globbing), do we know for certain that something like MyTest$NestedTests would be passed to the Provider? If not, I think the automatic inclusion of inner classes that you've described is the only option.

Would there ever be a need to execute a single test method within an inner (@nested) class? - Perhaps a pseudo-filter something like MyTest$NestedTests#thisTestDoesnt. I also realized that I didn't test the use of a period as a outerclass/innerclass separator. MyTest.NestedTests might work (I can try this when I get back to my home computer tonight).

[sbrannen]

Possibly related bug: https://stackoverflow.com/questions/47237611/check-that-junit-extension-throws-specific-exception/47982480?noredirect=1#comment86739467_47982480

[sbrannen]

FYI: I reproduced the bug discussed in the above SO comment.

Non-private, static, nested test classes are also not executed via the JUnit Platform Maven Surefire provider when executing mvn test.

The OP for the aforementioned SO post will likely create a separate issue for that, but I wanted to make sure it's mentioned here as well, so that we don't forget it.

[sbrannen]

FYI: the solution to this issue may be the same as for #1377.

Namely, Maven Surefire excludes all nested classes by default!

http://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#excludes

[sbrannen]

Tentatively slated for 5.2 GA in order to verify whether or not the fix for #1377 covers this use case as well.

[sbrannen]

in progress: investigating

[sbrannen]

Findings:

After adding a @Nested test class to FirstTest from the junit5-maven-consumer project and adding "no excludes" work-around discussed in #1377, we see that...

The following does not execute the nested test class.

./mvnw clean -Dtest=com.example.project.FirstTest test

However, the following does execute the nested test class. Note the trailing * after the fully qualified class name.

./mvnw clean -Dtest=com.example.project.FirstTest* test

After further analysis, I have determined that the above behavior is also experienced even if the "no excludes" work-around discussed in #1377 is not applied.

In summary, the "no excludes" work-around discussed in #1377 does not resolve this issue, and I am therefore moving this to 5.3.

[srdo]

@sormuras Sorry if I'm being dense, but I can't tell from the history. Was this fixed, or was it closed for another reason?

[sormuras]

I'm in the process of moving Surefire-related issues over to https://issues.apache.org/jira/projects/SUREFIRE/issues

[srdo]

Thanks.

[sormuras]

New location is https://issues.apache.org/jira/projects/SUREFIRE/issues/SUREFIRE-1633