Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug: Spoon fails to index modules when loaded from file system #4052

Closed
I-Al-Istannen opened this issue Jul 16, 2021 · 4 comments · Fixed by #4073
Closed

bug: Spoon fails to index modules when loaded from file system #4052

I-Al-Istannen opened this issue Jul 16, 2021 · 4 comments · Fixed by #4073
Labels

Comments

@I-Al-Istannen
Copy link
Collaborator

Problem

Spoon can not index a folder containing multiple modules but can index a ZIP containing the exact same data:

root
  `- module1
    `- module-info.java
  `- module2
    `- module-info.java

The exact exception is:

Exception in thread "main" java.lang.NullPointerException: Cannot invoke "org.eclipse.jdt.internal.compiler.lookup.AnnotationBinding.getAnnotationType()" because "annotationBinding" is null
	at org.eclipse.jdt.internal.compiler.problem.ProblemReporter.deprecatedSinceValue(ProblemReporter.java:1962)
	at org.eclipse.jdt.internal.compiler.problem.ProblemReporter.deprecatedType(ProblemReporter.java:1921)
	at org.eclipse.jdt.internal.compiler.ast.TypeReference.reportDeprecatedType(TypeReference.java:603)
	at org.eclipse.jdt.internal.compiler.ast.TypeReference.internalResolveType(TypeReference.java:564)
	at org.eclipse.jdt.internal.compiler.ast.TypeReference.resolveType(TypeReference.java:648)
	at org.eclipse.jdt.internal.compiler.ast.TypeReference.resolveType(TypeReference.java:644)
	at org.eclipse.jdt.internal.compiler.ast.TypeReference.resolveSuperType(TypeReference.java:616)
	at org.eclipse.jdt.internal.compiler.lookup.ClassScope.findSupertype(ClassScope.java:1665)
	at org.eclipse.jdt.internal.compiler.lookup.ClassScope.connectSuperInterfaces(ClassScope.java:1388)
	at org.eclipse.jdt.internal.compiler.lookup.ClassScope.connectTypeHierarchy(ClassScope.java:1450)
	at org.eclipse.jdt.internal.compiler.lookup.CompilationUnitScope.connectTypeHierarchy(CompilationUnitScope.java:367)
	at org.eclipse.jdt.internal.compiler.lookup.LookupEnvironment.completeTypeBindings(LookupEnvironment.java:518)
	at org.eclipse.jdt.internal.compiler.Compiler.internalBeginToCompile(Compiler.java:878)
	at org.eclipse.jdt.internal.compiler.Compiler.beginToCompile(Compiler.java:394)
	at spoon.support.compiler.jdt.TreeBuilderCompiler.buildUnits(TreeBuilderCompiler.java:60)
	at spoon.support.compiler.jdt.JDTBatchCompiler.getUnits(JDTBatchCompiler.java:256)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.buildUnits(JDTBasedSpoonCompiler.java:417)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.buildUnitsAndModel(JDTBasedSpoonCompiler.java:369)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.buildSources(JDTBasedSpoonCompiler.java:335)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.build(JDTBasedSpoonCompiler.java:116)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.build(JDTBasedSpoonCompiler.java:99)
	at spoon.Launcher.buildModel(Launcher.java:772)
	at de.ialistannen.javadocapi.indexing.Indexer.main(Indexer.java:45)

Cause

The JDT's main method parses the arguments and detects the presence of module-info.java files. For some reason (I have absolutely no clue why) it then assumes that all the input files are in this module and completly ignores the file paths and uses the first module it finds for every class. This then breaks JDT in interesting ways, as you can see in the stacktrace above.
A bit more concrete, this generates a pathtomod map like this:
grafik

Why it works with ZIP files

When using ZIP files, spoon will copy the ZIP file's contents to randomly named Name<number>.java files. Therefore, JDT will not recognize the module-info.java files and does not fall into the problematic codepath. This inspired the hack in the next section, as it seemed to work just fine ™️ with the zipfile and therefore without the module info file handling.

Possible solutions

To hack around this I currently do not pass module-info.java files as arguments to the JDT builder:

diff --git a/src/main/java/spoon/support/compiler/jdt/JDTBasedSpoonCompiler.java b/src/main/java/spoon/support/compiler/jdt/JDTBasedSpoonCompiler.java
index a39f5991..ff3a551b 100644
--- a/src/main/java/spoon/support/compiler/jdt/JDTBasedSpoonCompiler.java
+++ b/src/main/java/spoon/support/compiler/jdt/JDTBasedSpoonCompiler.java
@@ -7,6 +7,7 @@
  */
 package spoon.support.compiler.jdt;
 
+import java.util.stream.Collectors;
 import org.apache.commons.io.IOUtils;
 import org.eclipse.jdt.core.compiler.CategorizedProblem;
 import org.eclipse.jdt.core.compiler.IProblem;
@@ -400,7 +401,12 @@ public class JDTBasedSpoonCompiler implements spoon.SpoonModelBuilder {
                                complianceOptions.enablePreview();
                        }
                        AdvancedOptions advancedOptions = new AdvancedOptions().preserveUnusedVars().continueExecution().enableJavadoc();
-                       SourceOptions sourceOptions = new SourceOptions().sources(sourceFiles);
+                       SourceOptions sourceOptions = new SourceOptions().sources(
+                                       sourceFiles
+                                                       .stream()
+                                                       .filter(it -> !it.getName().contains("module-info"))
+                                                       .collect(Collectors.toList())
+                       );
                        args = new JDTBuilderImpl()
                                        .classpathOptions(classpathOptions)
                                        .complianceOptions(complianceOptions)

This works but as I don't know why we were passing them and why JDT treats them the way it does I don't expect it to be stable enough to use.

Maybe you have some ideas how to solve this.

@slarse slarse added the bug label Jul 16, 2021
@andrewbwogi
Copy link
Contributor

Hi @I-Al-Istannen, thanks for the report. I believe there is a bug in the JDT compiler. I'll look for a minimal bug reproduction and if it can be solved from Spoon for the time being.

@andrewbwogi
Copy link
Contributor

@I-Al-Istannen, I can reproduce the incorrect HashMap but not the exception. Can you share the code that makes Spoon crash?

@I-Al-Istannen
Copy link
Collaborator Author

I don't have a MWE, as I threw the jdk 16 source code at it and didn't trim it down. I could try to do that, if the JDK code is too much for you to work with.

The code is just

    System.out.println(heading("Configuring spoon"));
    Launcher launcher = new Launcher();
    launcher.getEnvironment().setShouldCompile(false);
    launcher.getEnvironment().disableConsistencyChecks();
    launcher.getEnvironment().setOutputType(OutputType.NO_OUTPUT);
    launcher.getEnvironment().setSpoonProgress(new ConsoleProcessLogger(launcher));
    launcher.getEnvironment().setCommentEnabled(true);
    launcher.getEnvironment().setComplianceLevel(15);
    for (String path : config.getResourcePaths()) {
      if (path.endsWith(".zip")) {
        launcher.addInputResource(new ZipFolder(new File(path)));
      } else {
        launcher.addInputResource(path);
      }
    }
    System.out.println("Spoon successfully configured\n");

    System.out.println(heading("Building spoon model"));
    CtModel model = launcher.buildModel();
    System.out.println("Model successfully built\n");

@I-Al-Istannen
Copy link
Collaborator Author

I trimmed down the JDK to 56 files which reproduce the error, but I couldn't really delete any more with trial and error:
Smaller.zip


Just not passing the modules is not an option, as other modular programs that require java classes don't quite like it apparently:

Exception in thread "main" java.lang.IllegalStateException: Module should be known
	at org.eclipse.jdt.internal.compiler.batch.CompilationUnit.module(CompilationUnit.java:138)
	at org.eclipse.jdt.internal.compiler.ast.CompilationUnitDeclaration.module(CompilationUnitDeclaration.java:848)
	at org.eclipse.jdt.internal.compiler.lookup.LookupEnvironment.buildTypeBindings(LookupEnvironment.java:471)
	at org.eclipse.jdt.internal.compiler.Compiler.internalBeginToCompile(Compiler.java:855)
	at org.eclipse.jdt.internal.compiler.Compiler.beginToCompile(Compiler.java:394)
	at spoon.support.compiler.jdt.TreeBuilderCompiler.buildUnits(TreeBuilderCompiler.java:60)
	at spoon.support.compiler.jdt.JDTBatchCompiler.getUnits(JDTBatchCompiler.java:256)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.buildUnits(JDTBasedSpoonCompiler.java:423)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.buildUnitsAndModel(JDTBasedSpoonCompiler.java:370)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.buildSources(JDTBasedSpoonCompiler.java:336)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.build(JDTBasedSpoonCompiler.java:117)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.build(JDTBasedSpoonCompiler.java:100)
	at spoon.Launcher.buildModel(Launcher.java:772)
	at de.ialistannen.javadocapi.indexing.Indexer.main(Indexer.java:45)

I didn't have any trouble after dropping the hacky patch above - that seems to only be triggered by the JDK itself.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants