bug: Spoon fails to index modules when loaded from ZIP

[I-Al-Istannen]

Problem statement

It would be kind of funny if it wasn't so sad, but after #4052 indexing no longer works correctly with a ZIP as input source... Indexing from the file system works though 🙃

More elaborate problem statement

My JavadocAPI refuses to find Javadoc for quite a few entries in JDK 16 when it is indexed from a ZIP file. The element is there but it finds no comment. This only happens if I include a substantial part of the JDK in the zip.

I would love to test my MWE from the other bug but that crashes Spoon during model building:

==== Building spoon model ====
Starting phase COMPILE
Phase COMPILE done! Discovered Classes: 49
Starting phase COMMENT
Phase COMMENT done! Discovered Classes: 49
Starting phase MODEL
Exception in thread "main" java.lang.NullPointerException: Cannot invoke "org.eclipse.jdt.internal.compiler.lookup.AnnotationBinding.getAnnotationType()" because "annotation" is null
	at spoon.support.compiler.jdt.JDTTreeBuilderQuery.hasAnnotationWithType(JDTTreeBuilderQuery.java:178)
	at spoon.support.compiler.jdt.ParentExiter.scanCtElement(ParentExiter.java:156)
	at spoon.reflect.visitor.CtInheritanceScanner.visitCtField(CtInheritanceScanner.java:593)
	at spoon.support.reflect.declaration.CtFieldImpl.accept(CtFieldImpl.java:53)
	at spoon.reflect.visitor.CtInheritanceScanner.scan(CtInheritanceScanner.java:181)
	at spoon.support.compiler.jdt.ContextBuilder.exit(ContextBuilder.java:132)
	at spoon.support.compiler.jdt.JDTTreeBuilder.endVisit(JDTTreeBuilder.java:640)
	at org.eclipse.jdt.internal.compiler.ast.SingleMemberAnnotation.traverse(SingleMemberAnnotation.java:76)
	at org.eclipse.jdt.internal.compiler.ast.FieldDeclaration.traverse(FieldDeclaration.java:375)
	at org.eclipse.jdt.internal.compiler.ast.TypeDeclaration.traverse(TypeDeclaration.java:1699)
	at org.eclipse.jdt.internal.compiler.ast.CompilationUnitDeclaration.traverse(CompilationUnitDeclaration.java:827)
	at org.eclipse.jdt.internal.compiler.ast.CompilationUnitDeclaration.traverse(CompilationUnitDeclaration.java:788)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.traverseUnitDeclaration(JDTBasedSpoonCompiler.java:480)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.lambda$buildModel$0(JDTBasedSpoonCompiler.java:437)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.forEachCompilationUnit(JDTBasedSpoonCompiler.java:464)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.buildModel(JDTBasedSpoonCompiler.java:435)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.buildUnitsAndModel(JDTBasedSpoonCompiler.java:372)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.buildSources(JDTBasedSpoonCompiler.java:335)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.build(JDTBasedSpoonCompiler.java:116)
	at spoon.support.compiler.jdt.JDTBasedSpoonCompiler.build(JDTBasedSpoonCompiler.java:99)
	at spoon.Launcher.buildModel(Launcher.java:781)
	at de.ialistannen.javadocapi.indexing.Indexer.main(Indexer.java:62)

The MWE works when I unzip it however...

CC: @andrewbwogi

---

EDIT:
A lead could be this

result of the JDT "Main" class trying to extract the module info by parsing the compilation unit. This works when the file is named correctly without the random temporary file numbers.

I "fixed" this locally by doing:

diff --git a/src/main/java/spoon/compiler/builder/SourceOptions.java b/src/main/java/spoon/compiler/builder/SourceOptions.java
index f602d34e..b9e2d27c 100644
--- a/src/main/java/spoon/compiler/builder/SourceOptions.java
+++ b/src/main/java/spoon/compiler/builder/SourceOptions.java
@@ -7,14 +7,15 @@
  */
 package spoon.compiler.builder;
 
-import org.apache.commons.io.IOUtils;
-import spoon.compiler.SpoonFile;
-
 import java.io.File;
 import java.io.FileOutputStream;
 import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
 import java.util.Arrays;
 import java.util.List;
+import org.apache.commons.io.IOUtils;
+import spoon.compiler.SpoonFile;
 
 public class SourceOptions<T extends SourceOptions<T>> extends Options<T> {
 	public SourceOptions() {
@@ -45,21 +46,46 @@ public class SourceOptions<T extends SourceOptions<T>> extends Options<T> {
 			args.add(".");
 			return myself;
 		}
-		for (SpoonFile source : sources) {
-			if (source.isActualFile()) {
-				args.add(source.toString());
-			} else {
+		try {
+			Path path = Files.createTempDirectory("spoon-container");
+
+			Runtime.getRuntime().addShutdownHook(new Thread(() -> {
 				try {
-					File file = File.createTempFile(source.getName(), ".java");
-					file.deleteOnExit();
-					try (FileOutputStream fileOutputStream = new FileOutputStream(file)) {
-						IOUtils.copy(source.getContent(), fileOutputStream);
-					}
-					args.add(file.toString());
+					Files.walk(path).sorted(Comparator.reverseOrder()).forEach(
+							path1 -> {
+								try {
+									Files.deleteIfExists(path1);
+								} catch (IOException e) {
+									e.printStackTrace();
+								}
+							});
 				} catch (IOException e) {
-					throw new RuntimeException(e.getMessage(), e);
+					e.printStackTrace();
+				}
+			}));
+
+			for (SpoonFile source : sources) {
+				if (source.isActualFile()) {
+					args.add(source.toString());
+				} else {
+					try {
+						String name = source.getName();
+						if (name.startsWith("/")) {
+							name = name.substring(1);
+						}
+						File file = path.resolve(name).toFile();
+						file.getParentFile().mkdirs();
+						try (FileOutputStream fileOutputStream = new FileOutputStream(file)) {
+							IOUtils.copy(source.getContent(), fileOutputStream);
+						}
+						args.add(file.toString());
+					} catch (IOException e) {
+						throw new RuntimeException(e.getMessage(), e);
+					}
 				}
 			}
+		} catch (IOException e) {
+			e.printStackTrace();
 		}
 		return myself;
 	}

Which names the files correctly after extracting them from the ZIP.