Classpath scanning does not work in tests using the module path [SPR-16977]

[spring-projects-issues]

Andy Wilkinson opened SPR-16977 and commented

It appears the classpath scanning doesn't work when Surefire launches the JVM configured to use the module path. target/classes is placed on the module path and target/test-classes is patched into this module but classpath scanning only finds classes in target/test-classes.

I have attached a minimal sample that should reproduce the problem when built (mvn test) with Java 10. The sysout from the test should show that only the test class has been found:

[INFO] Running com.example.ScanningTest
[file [/Users/awilkinson/dev/temp/module-path-scanning/target/test-classes/com/example/ScanningTest.class]]

To my rather untrained eye, building with -X and examining the arguments that Surefire uses to launch the forked JVM (in target/surefire) suggests that Surefire's configuration of the JVM is correct.

When the sample is modified to work with Java 8 (remove module-info.java and change the compiler plugin configuration to remove <release>10</release>) the class in target/classes is also found:

[INFO] Running com.example.ScanningTest
[file [/Users/awilkinson/dev/temp/module-path-scanning/target/test-classes/com/example/ScanningTest.class], file [/Users/awilkinson/dev/temp/module-path-scanning/target/classes/com/example/One.class]]

---

Affects: 5.0.7

Reference URL: spring-projects/spring-boot#13581

Attachments:

• module-path-scanning.zip (3.66 kB)

Issue Links:

• Compatibility with JDK 11 [SPR-16391] #20937 Compatibility with JDK 11

2 votes, 7 watchers

[spring-projects-issues]

Juergen Hoeller commented

This all depends on ClassLoader.getResources results for the given base package. The module system possibly only exposes the root URL for the patched part there? In any case, debugging ClassLoader.getResources results in both scenarios would help here...

[spring-projects-issues]

Andy Wilkinson commented

On closer inspection, this appears to be a bug in the JDK. With the test changed to the following:

@Test
public void scanningTest() throws Exception {
    Enumeration<URL> resourceUrls = getClass().getClassLoader().getResources("com/example");
    while (resourceUrls.hasMoreElements()) {
        System.out.println(resourceUrls.nextElement());
    }
    PathMatchingResourcePatternResolver resolver = new PathMatchingResourcePatternResolver();
        System.out.println(Arrays.toString(resolver.getResources("classpath*:com/example/**/*.class")));
}

It produces output similar to this:

[INFO] Running com.example.ScanningTest
file:/Users/awilkinson/dev/temp/module-path-scanning/target/test-classes/com/example/
file:/Users/awilkinson/dev/temp/module-path-scanning/target/test-classes/com/example
[file [/Users/awilkinson/dev/temp/module-path-scanning/target/test-classes/com/example/ScanningTest.class]]
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.094 s - in com.example.ScanningTest

Note the two (slightly different) URLs that are both pointing to target/test-classes. This explains why no classes in target/classes are found.

[spring-projects-issues]

Juergen Hoeller commented

Looks like a bug in the ClassLoader implementation indeed. They might not have fully tested this in module patch scenarios...

Probably worth reporting to OpenJDK. Aside from a proper fix, maybe there's a workaround to discover in time for our GA still.

[spring-projects-issues]

Andy Wilkinson commented

I've reported a bug. It's in internal review at the moment (ID 9055803). I'll comment again here if it makes it out the other end and becomes public. For reference, this is the test case that I provided with the bug report. Unlike the sample attached to this issue, it does not use Spring Framework.

[spring-projects-issues]

Andy Wilkinson commented

The bug is now public but has been closed as not an issue. The situation's now being discussed in this thread on the core libs dev OpenJDK mailing list.

[spring-projects-issues]

Andy Wilkinson commented

The thread on the mailing list seems to have reached a conclusion. The following recommendation came from Alan Bateman:

With modules then it should be looking at the modules in the boot layer and using ModuleReader to get the contents. It can use the value of java.class.path to scan the class path.

[spring-projects-issues]

Thomas Kratz commented

Are there any plans to resolve this like recommended? 

Would doFindAllClassPathResources(String path) be the point to put that? I don't clearly understand that suggestion, but maybe I could invest some time over the holidays.

I would still think this is a JDK issue.

 

[wimdeblauwe]

Is there a workaround for this? I have a JavaFx project that I added Spring Boot to which works really great, but I am unable to run my @JsonTest.

As a workaround, I need to manually specify the application class + the json deserializers I want to test:

@JsonTest
class MyJsonDeserializerTest {
   ...
}

needs to become:

@JsonTest
@ContextConfiguration(classes = {MyApplication.class, MyJsonDeserializerTest.TestConfig.class})
class MyJsonDeserializerTest {
   ...

    @TestConfiguration
    static class TestConfig {
        @Bean
        public MyJsonDeserializer deserializer() {
            return new MyJsonDeserializer();
        }
    }

}

(Note: It seems I only need to do this for Maven, IntelliJ seems to run the test fine, but I have to assume that IntelliJ is somehow "cheating" given this issue and the related spring-projects/spring-boot#13581 )

[eiswind]

Another year has passed.
To me this still is a showstopper.

[tgolden-andplus]

I have the opposite problem currently. Failsafe works just fine, apparently because they point to the packaged JAR instead of the exploded target/classes directory. However, IntelliJ wants to use the exploded directory when running integration tests in the IDE, so this fails unless I uncheck the module-path option (which I have to do for every run configuration separately).

Additionally, I can make Failsafe break the same way as the IDE by forcing it to use the exploded directory (via <classesDirectory>${project.build.outputDirectory}</classesDirectory>), so there's something magic about being in a JAR versus hanging around on the filesystem -- but I'm not clear on what.

Is this truly a JDK issue or a Spring issue? It doesn't seem that the core JDK team plans on changing behavior, thus it seems incumbent on the Spring team to work around this somehow (or at the very least, document that component scanning will break unexpectedly when using exploded classes versus packaged in JAR classes....)

[tomdw]

@jhoeller @wilkinsona could this be fixed inside spring by making the component scanning code be more module path aware? Especially for running spring tests on module path using the surefire plugin or in intellij this is a problem.

[PeterMylemans]

@tgolden-andplus I'm having the same issue.

For future reference; the only work around I found was to uncheck "use module path" in Templates > JUnit in your Run/Debug Generations to make sure it is unchecked for every new junit run config you create.

[manosbatsis]

For me the workaround was using a org.springframework.core.io.ClassPathResource instead of ClassLoader.getResource

[tomdw]

@manosbatsis and how dit you make sure that spring uses this ClassPathResource for its component scanning?

@tgolden-andplus problem with that workaround is that you then no longer have the module checking which is important. Also stops working when using e.g. serviceloader to load services from other modules in your code under test

Probably the only fix is a fix in the code of spring itself, solving the way it does component scanning to handle modules better.

[manosbatsis]

@tomdw to be clear, i had trouble when trying to "manually" load a classpath resource from within tests using ClassLoader.getResource. After a search landed me here, i figured out a workaround was needed to bypass the bug and ClassPathResource did the trick for me, so i thought it might be useful.

[retheesh-mr]

I am facing similar issue when writing junit5 cases with spring boot and java 11. As the jdk bug raised is in resolved status, what is further action on this? How can this be resolved?
It seems module-info cannot be used together with junit and spring due to this issue. As it is blocking for me in my project I have no other way than to fall back to java 8 version.

[kopper]

I had the same (or very similar) issue running tests with maven-failsafe-plugin 3.0.0-M5 on AdoptOpen JDK 11. The work around is to disable using module path in plugin's configuration
<useModulePath>false</useModulePath>

[oscarhaglund]

I should mention that I am also suffering from this issue and it would really be appreciated if this could be solved by Spring if it is not going to be solved by Java itself.

[prakash0409]

We upgraded to a new version of spring boot 2.3.4 and all our integration tests started failing. All our integration test cases were earlier working on AdoptOpen JDK 11 with the older version. I guess this should be an issue at Spring end. As mentioned above the classpath scanning is only finding only "target/test-classes".

I was able to use the workaround as suggested by @kopper

[msche]

FYI, I had similar classpath issues when I introduced java modules in my project and executed the Spring Boot unit tests.

07:28:35.820 [main] ERROR - o.s.test.context.TestContextManager - prepareTestInstance - Caught exception while allowing TestExecutionListener [org.springframework.test.context.support.DependencyInjectionTestExecutionListener@63e5b8aa] to prepare test instance [...]
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name '...': Unsatisfied dependency expressed through field '...'; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No qualifying bean of type '...' available: expected at least 1 bean which qualifies as autowire candidate. Dependency annotations: {@org.springframework.beans.factory.annotation.Autowired(required=true)}

I eventually got the test executed successfully by including within the configuration of SureFire version 2.22.x <forkCount>0</forkCount>.

[barbarosalp]

Is there any proper solution, we are also stuck with this?

[OleksandrGavryliukTR]

FYI, I had similar classpath issues when I introduced java modules in my project and executed the Spring Boot unit tests.

07:28:35.820 [main] ERROR - o.s.test.context.TestContextManager - prepareTestInstance - Caught exception while allowing TestExecutionListener [org.springframework.test.context.support.DependencyInjectionTestExecutionListener@63e5b8aa] to prepare test instance [...]
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name '...': Unsatisfied dependency expressed through field '...'; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No qualifying bean of type '...' available: expected at least 1 bean which qualifies as autowire candidate. Dependency annotations: {@org.springframework.beans.factory.annotation.Autowired(required=true)}

I eventually got the test executed successfully by including within the configuration of SureFire version 2.22.x <forkCount>0</forkCount>.

This has fixed it for me. With a warning:
[WARNING] useSystemClassloader setting has no effect when not forking
Which must be removing the cause of this problem.

[sbrannen]

Disclaimer: This is in no way production ready code. This is 100% "proof of concept" code.

Now, having said that... with an updated version of the original example application from @wilkinsona (now using Java 17, Maven 3.8.5, Maven Surefire 3.0.0-M6, Spring Framework 5.3.20, and JUnit Jupiter 5.8.2), I have come up with the following.

class ScanningTest {

	@Test
	void scanningTest() throws Exception {
		PathMatchingResourcePatternResolver resolver = new PathMatchingResourcePatternResolver();
		System.err.println(Arrays.toString(resolver.getResources("classpath*:com/example/**/*.class")));

		System.err.println("------------------------------------------------");

		String moduleName = getClass().getModule().getName();
		List<String> resourceNames = streamResolvedModules(Predicate.isEqual(moduleName))//
				.map(ResolvedModule::reference)//
				.map(moduleReference -> scanForNames(moduleReference, "^com/example/.+\\.class$"))//
				.flatMap(List::stream)//
				.toList();

		resourceNames.forEach(System.err::println);

		System.err.println("------------------------------------------------");

		resourceNames.forEach(resouceName -> {
			String className = ClassUtils.convertResourcePathToClassName(resouceName);
			className = className.substring(0, className.length() - ".class".length());
			try {
				Class<?> clazz = ClassUtils.forName(className, getClass().getClassLoader());
				System.err.println(clazz);
			}
			catch (Exception ex) {
				throw new RuntimeException("Failed to load class %s".formatted(className), ex);
				// ex.printStackTrace(System.err);
			}
		});

		System.err.println("------------------------------------------------");

		List<URI> resources = streamResolvedModules(Predicate.isEqual(moduleName))//
				.map(ResolvedModule::reference)//
				.map(moduleReference -> scanForResources(moduleReference, "^com/example/.+\\.class$"))//
				.flatMap(List::stream)//
				.toList();

		resources.forEach(resource -> {
			try {
				System.err.println(resource.toURL());
			}
			catch (MalformedURLException ex) {
				ex.printStackTrace();
			}
		});
	}

	private Stream<ResolvedModule> streamResolvedModules(Predicate<String> moduleNamePredicate) {
		ModuleLayer layer = getClass().getModule().getLayer();
		if (layer == null) {
			layer = ModuleLayer.boot();
		}
		return layer.configuration().modules().stream()//
				.filter(module -> moduleNamePredicate.test(module.name()));
	}

	private List<String> scanForNames(ModuleReference reference, String regex) {
		try (ModuleReader reader = reference.open()) {
			try (Stream<String> names = reader.list()) {
				return names.filter(name -> name.matches(regex)).toList();
			}
		}
		catch (IOException ex) {
			throw new UncheckedIOException("Failed to read contents of " + reference, ex);
		}
	}

	private List<URI> scanForResources(ModuleReference reference, String regex) {
		try (ModuleReader reader = reference.open()) {
			try (Stream<String> names = reader.list()) {
				return names.filter(name -> name.matches(regex))//
						.map(name -> {
							try {
								return reader.find(name);
							}
							catch (IOException ex) {
								ex.printStackTrace();
							}
							return Optional.<URI> empty();
						})//
						.filter(Optional::isPresent)//
						.map(Optional::get).toList();
			}
		}
		catch (IOException ex) {
			throw new UncheckedIOException("Failed to read contents of " + reference, ex);
		}
	}

}

The output of running that is:

[file [/Users/sbrannen/source/spring-issues/module-path-scanning/target/test-classes/com/example/ScanningTest.class]]
------------------------------------------------
com/example/One.class
com/example/ScanningTest.class
------------------------------------------------
class com.example.One
class com.example.ScanningTest
------------------------------------------------
file:/Users/sbrannen/source/spring-issues/module-path-scanning/target/classes/com/example/One.class
file:/Users/sbrannen/source/spring-issues/module-path-scanning/target/test-classes/com/example/ScanningTest.class

The first output demonstrates the issue: PathMatchingResourcePatternResolver only finds ScanningTest.class and not One.class.

The final output demonstrates that we can find resources (as instances of URI) using the Module APIs and successfully find both ScanningTest.class and One.class.

Kudos to @sormuras for providing inspirational use of the java.lang.module APIs in JUnit 5!

---

Where to go from here?

We may introduce a new ModulePathResource as a companion to ClassPathResource, where the logic in ModulePathResource is similar to the logic in the above proof of concept.

We may then introduce additional support in PathMatchingResourcePatternResolver that resolves ModulePathResource instances when appropriate.

[sbrannen]

Current work on this issue can be viewed in the following feature branch.

https://github.com/sbrannen/spring-framework/commits/module-path-scanning

[sbrannen]

Hi everybody,

Thanks for all of the feedback over the years!

I am now closing this issue since it has been superseded by the following issues.

• Support module path scanning for "classpath*:" resource prefix #28506
• Introduce ModuleResource for resources loaded from a given Module #28507

Please follow those issues for additional updates on module path scanning and resource support.

[sbrannen]

If you've been following this issue, I'm happy to let you know that #28506 has been resolved for inclusion in 6.0 M5.

In the interim, feel free to try out 6.0 snapshots with modular testing using Maven.

I've created a new repository for demonstrating the use of the Spring Framework with the Java Module System: https://github.com/sbrannen/spring-module-system

That repository currently contains a maven-surefire-patched-module project which demonstrates support for @ComponentScan in a patched module using Maven Surefire.