From 9a9a6e65a7007c7500af7684202fb1b35376c12d Mon Sep 17 00:00:00 2001 From: Jaroslav Tulach Date: Mon, 27 Nov 2023 09:37:51 +0100 Subject: [PATCH 1/2] Unifying debugging information (#8370) --- docs/debugger/README.md | 4 +- docs/debugger/mixed-debugging.md | 47 +----------- docs/debugger/runtime-debugging.md | 116 ++++++++++++++++++++++++----- tools/enso4igv/README.md | 2 + 4 files changed, 102 insertions(+), 67 deletions(-) diff --git a/docs/debugger/README.md b/docs/debugger/README.md index 8536eb1b95a4..bde536bb656c 100644 --- a/docs/debugger/README.md +++ b/docs/debugger/README.md @@ -17,7 +17,5 @@ used by Enso, broken up as follows: Enso code using Chrome devtools. - [**Debug adapter protocol:**](./dap.md) A guide how to debug Enso code using VSCode. -- [**Debugging Enso and Java code at once:**](./mixed-debugging.md) A - step-by-step guide how to debug both Enso and Java code in a single debugger. - [**Debugging Engine (Runtime) only Java code:**](./runtime-debugging.md) A - guide how to debug the internal Engine Java code. + guide how to debug Enso code & the internal Java code. diff --git a/docs/debugger/mixed-debugging.md b/docs/debugger/mixed-debugging.md index 48f0f4107f79..a4ea37855abe 100644 --- a/docs/debugger/mixed-debugging.md +++ b/docs/debugger/mixed-debugging.md @@ -1,48 +1,3 @@ # Debugging Enso and Java Code at Once -Enso libraries are written in a mixture of Enso code and Java libraries. -Debugging both sides (the Java as well as Enso code) is possible with a decent -IDE. - -Get [NetBeans](http://netbeans.apache.org) version 13 or newer or -[VS Code with Apache Language Server extension](https://cwiki.apache.org/confluence/display/NETBEANS/Apache+NetBeans+Extension+for+Visual+Studio+Code) -and _start listening on port 5005_ with _Debug/Attach Debugger_ or by specifying -following debug configuration in VSCode: - -```json -{ - "name": "Listen to 5005", - "type": "java+", - "request": "attach", - "listen": "true", - "hostName": "localhost", - "port": "5005" -} -``` - -Then it is just about executing following Sbt command which builds CLI version -of the engine, launches it in debug mode and passes all other arguments to the -started process: - -```bash -sbt:enso> runEngineDistribution --debug --run ./test/Tests/src/Data/Numbers_Spec.enso -``` - -Alternatively you can pass in special JVM arguments when launching the -`bin/enso` launcher: - -```bash -enso$ JAVA_OPTS=-agentlib:jdwp=transport=dt_socket,server=n,address=5005 ./built-distribution/enso-engine-*/enso-*/bin/enso --run ./test/Tests/src/Data/Numbers_Spec.enso -``` - -As soon as the debuggee connects and the Enso language starts - choose the -_Toggle Pause in GraalVM Script_ button in the toolbar: - -![NetBeans Debugger](https://user-images.githubusercontent.com/26887752/209614191-b0513635-819b-4c64-a6f9-9823b90a1513.png) - -and your execution shall stop on the next `.enso` line of code. This mode allows -to debug both - the Enso code as well as Java code. The stack traces shows a -mixture of Java and Enso stack frames by default. Right-clicking on the thread -allows one to switch to plain Java view (with a way more stack frames) and back. -Analyzing low level details as well as Enso developer point of view shall be -simple with such tool. +Desribed at [Runtime Debugging](runtime-debugging.md). diff --git a/docs/debugger/runtime-debugging.md b/docs/debugger/runtime-debugging.md index 6ab9ea242737..a170af1d8c27 100644 --- a/docs/debugger/runtime-debugging.md +++ b/docs/debugger/runtime-debugging.md @@ -1,37 +1,117 @@ -# Runtime (Engine) debugging +# High & Low Runtime Debugging -This section explains how to debug various parts of the Engine. By Engine, we -mean all the Java code located in the `runtime` SBT project, in `engine` -directory. +This document describes how to debug **everything** - e.g. how to debug the +_Java code_ that powers the Enso _engine & interpreter_ as well as Enso standard +_libraries Java code_ (used to interact with the operating system via Java +APIs). At the same thetime the document describes how to _map that information_ +onto the execution of the actual _Enso source code_ being interpreted. -## Debugging source file evaluation +## Debugging Single Source File -This subsection provides a guide how to debug a single Enso source file -evaluation. To evaluate a single source file, we use the _Engine distribution_ -built with `buildEngineDistribution` command. Both of the following two options -starts the JVM in a debug mode. After the JVM is started in a debug mode, simply -attach the debugger to the JVM process at the specified port. +To analyze what is happening in the interpreter, we often want to debug +excecution of a _single Enso file_. -The first option is to invoke the engine distribution from SBT shell with: +Get started by building the _Engine distribution_ built with +`sbt buildEngineDistribution` command. Then configure your IDE to understand the +_engine Java & Scala sources_ which are mostly located in the +[engine subdirectory](https://github.com/enso-org/enso/tree/develop/engine). The +sources are known to be understandable by: + +- VSCode - with + [Enso Extension described herein](../../tools/enso4igv/README.md) +- IntelliJ +- IGV - with [extension described herein](../../tools/enso4igv/IGV.md) +- [NetBeans](http://netbeans.apache.org) - with the same + [extension as IGV](../../tools/enso4igv/IGV.md) is using + +There are two ways to start JVM in a debug mode. The first one is fully +integrated into the `sbt` build system. It builds the engine (in case there were +any changes) and then starts the JVM in debug mode trying to attach to port +`5005`: ```sh sbt:enso> runEngineDistribution --debug --run ./test/Tests/src/Data/Numbers_Spec.enso ``` -The second options is to pass in special JVM arguments when launching the -`bin/enso` from the engine distribution: +The second options gives one a complete control as it launches everything from a +command line. By specifying `JAVA_OPTS` environment variable one influences the +special JVM arguments when launching the `bin/enso` from the engine +distribution: ```bash enso$ JAVA_OPTS=-agentlib:jdwp=transport=dt_socket,server=n,address=5005 ./built-distribution/enso-engine-*/enso-*/bin/enso --run ./test/Tests/src/Data/Numbers_Spec.enso ``` -### Tips and tricks +Both of the approaches launch the JVM in a _debug mode_. Once the JVM is +started, simply attach the debugger to the JVM process at the specified port +(usually `5005` is used). + +### Attaching from VSCode + +First of all make sure your VSCode contains the +[Enso supporting extension](https://marketplace.visualstudio.com/items?itemName=Enso.enso4vscode). +Follow the [instructions provided herein](../../tools/enso4igv/README.md) to +install everything that's needed. + +Once the +[Enso extension](https://marketplace.visualstudio.com/items?itemName=Enso.enso4vscode) +is installed, open the root of Enso repository as workspace and select _"Listen +to 5005"_ debug configuration: + +![Listen to 5005](https://github.com/enso-org/enso/assets/26887752/1874bcb1-cf8b-4df4-92d8-e7fb57e1b17a) + +Once the connection with the Enso JVM is made, debug the Java code as usual. + +### Mapping Java and Enso Code + +The _Enso interpreter_ maintains a mapping between its Java code and appropriate +_Enso Code_ internally. It shall be no problem to use that information in the +debugger. All the mentioned IDEs (alas except IntelliJ) support such _mixed +debugging_ of Java and Enso code. + +Put a breakpoint into `.enso` file and after a while the breakpoint is hit and +one can inspect variables, step over the _Enso statements_ and more... + +![Breakpoint in Enso](https://github.com/enso-org/enso/assets/26887752/54ae4126-f77a-4463-9647-4dd3a5f83526) + +...as one can seamlessly switch to debugging on the Enso interpreter itself! One +can place breakpoint into Java class like `PanicException.java` and continue +debugging with `F5`: + +![Breakpoint in Java](https://github.com/enso-org/enso/assets/26887752/db3fbe4e-3bb3-4d4a-bb2a-b5039f716c85) + +Should one ever want to jump back from Java to Enso one can use the _"Pause in +GraalVM Script"_ action. Select it and continue with `F5` - as soon as the code +reaches a statement in Enso, it stops: + +![Pause in GraalVM](https://github.com/enso-org/enso/assets/26887752/98eb0bb7-48c2-4208-9d9a-5b8bacc99de2) + +Similar behavior to _VSCode_ is available in _IGV_ and _NetBeans_. Understanding +both the _engine Java code_ as well as _Enso Code_ shall be pretty simple with +these tools. For example _NetBeans_ offers _Toggle Pause in GraalVM Script_ +button in the toolbar: + +![NetBeans Debugger](https://user-images.githubusercontent.com/26887752/209614191-b0513635-819b-4c64-a6f9-9823b90a1513.png) + +and your execution shall stop on the next `.enso` line of code. + +These advanced developer tools allow one to debug both - the _Enso code_ as well +as _Java code_. The stack traces shows a _mixture of Java and Enso stack frames_ +by default. Right-clicking on the thread allows one to switch to plain Java view +(with a way more stack frames) and back. Analyzing low level details as well as +Enso developer point of view shall be simple with such tool. + +### Tips and Tricks (for poor IntelliJ users) -There is no simple mapping of the Enso source code to the engine's Java code, so -if you try to debug a specific expression, it might be a bit tricky. However, -the following steps should help you to skip all the irrelevant code and get to -the code you are interested in: +Finding the mapping of the Enso source code to the engine's Java code in +_IntelliJ_ isn't easy. Trying to find out how to debug a specific expression is +way more tricky than in case of _VSCode_, _IGV_ or _NetBeans_. However, if you +really want to stick with _IntelliJ_ as your only tool, following steps may help +you to skip all the irrelevant code and get to the code you are interested in: +- To get the mapping to Enso source code, evaluate the following expression in + the Java debugger: `this.getRootNode().getSourceSection()`. Note that this, + obviously, works only when the debugger is stopped in some Truffle node code. - To debug a method called `foo`, put a breakpoint in `org.enso.interpreter.node.ClosureRootNode#execute` with a condition on `this.name.contains("foo")` diff --git a/tools/enso4igv/README.md b/tools/enso4igv/README.md index f9d7e712d853..ec0853f03135 100644 --- a/tools/enso4igv/README.md +++ b/tools/enso4igv/README.md @@ -72,6 +72,8 @@ reaches a statement in Enso, it stops: ![Pause in GraalVM](https://github.com/enso-org/enso/assets/26887752/98eb0bb7-48c2-4208-9d9a-5b8bacc99de2) +Read more on [Enso & Java Debugging](../../docs/debugger/runtime-debugging.md) + ## Building VSCode Extension To build this VSCode extension and obtain _Enso_ syntax coloring as well as From 893965ed5cf5f9b950a21b8c6be4c47d6d1c51fe Mon Sep 17 00:00:00 2001 From: Jaroslav Tulach Date: Mon, 27 Nov 2023 10:28:12 +0100 Subject: [PATCH 2/2] 3% speedup with LazyMap and MetadataStorage (#8359) --- .../pass/analyse/PassPersistance.java | 30 ++++ .../enso/compiler/phase/ImportResolver.scala | 45 ++++-- .../org/enso/compiler/core/ir/IrLazyMap.java | 85 +++++++++++ .../enso/compiler/core/ir/IrPersistance.java | 46 ++---- .../enso/compiler/core/ir/IrLazyImMap.scala | 37 +++++ .../compiler/core/ir/MetadataStorage.scala | 14 +- .../enso/compiler/core/IrPersistanceTest.java | 55 ++++++- .../runtime/TruffleCompilerContext.java | 44 +++--- .../persist/impl/PersistableProcessor.java | 136 ++++++++++-------- .../java/org/enso/persist/PerInputImpl.java | 2 +- test/Benchmarks/src/Main.enso | 2 +- test/Benchmarks/src/Startup.enso | 52 ------- test/Benchmarks/src/Startup/Empty_World.enso | 1 + test/Benchmarks/src/Startup/Hello_World.enso | 3 + test/Benchmarks/src/Startup/Startup.enso | 70 +++++++++ 15 files changed, 445 insertions(+), 177 deletions(-) create mode 100644 engine/runtime-parser/src/main/java/org/enso/compiler/core/ir/IrLazyMap.java create mode 100644 engine/runtime-parser/src/main/scala/org/enso/compiler/core/ir/IrLazyImMap.scala delete mode 100644 test/Benchmarks/src/Startup.enso create mode 100644 test/Benchmarks/src/Startup/Empty_World.enso create mode 100644 test/Benchmarks/src/Startup/Hello_World.enso create mode 100644 test/Benchmarks/src/Startup/Startup.enso diff --git a/engine/runtime-compiler/src/main/java/org/enso/compiler/pass/analyse/PassPersistance.java b/engine/runtime-compiler/src/main/java/org/enso/compiler/pass/analyse/PassPersistance.java index e4d66da2c0b8..e3a0875a743a 100644 --- a/engine/runtime-compiler/src/main/java/org/enso/compiler/pass/analyse/PassPersistance.java +++ b/engine/runtime-compiler/src/main/java/org/enso/compiler/pass/analyse/PassPersistance.java @@ -3,10 +3,22 @@ import java.io.IOException; import org.enso.compiler.pass.analyse.AliasAnalysis.Graph; import org.enso.compiler.pass.resolve.DocumentationComments; +import org.enso.compiler.pass.resolve.DocumentationComments$; +import org.enso.compiler.pass.resolve.ExpressionAnnotations$; import org.enso.compiler.pass.resolve.FullyQualifiedNames; +import org.enso.compiler.pass.resolve.FullyQualifiedNames$; +import org.enso.compiler.pass.resolve.GenericAnnotations$; +import org.enso.compiler.pass.resolve.GlobalNames$; import org.enso.compiler.pass.resolve.IgnoredBindings; +import org.enso.compiler.pass.resolve.IgnoredBindings$; +import org.enso.compiler.pass.resolve.MethodCalls$; +import org.enso.compiler.pass.resolve.MethodDefinitions$; import org.enso.compiler.pass.resolve.ModuleAnnotations; +import org.enso.compiler.pass.resolve.ModuleAnnotations$; +import org.enso.compiler.pass.resolve.Patterns$; +import org.enso.compiler.pass.resolve.TypeNames$; import org.enso.compiler.pass.resolve.TypeSignatures; +import org.enso.compiler.pass.resolve.TypeSignatures$; import org.enso.persist.Persistable; import org.enso.persist.Persistance; import org.openide.util.lookup.ServiceProvider; @@ -30,6 +42,24 @@ @Persistable(clazz = FullyQualifiedNames.FQNResolution.class, id = 1128) @Persistable(clazz = FullyQualifiedNames.ResolvedLibrary.class, id = 1129) @Persistable(clazz = FullyQualifiedNames.ResolvedModule.class, id = 1130) +@Persistable(clazz = AliasAnalysis$.class, id = 1201) +@Persistable(clazz = BindingAnalysis$.class, id = 1202) +@Persistable(clazz = CachePreferenceAnalysis$.class, id = 1203) +@Persistable(clazz = DataflowAnalysis$.class, id = 1204) +@Persistable(clazz = GlobalNames$.class, id = 1205) +@Persistable(clazz = IgnoredBindings$.class, id = 1206) +@Persistable(clazz = Patterns$.class, id = 1207) +@Persistable(clazz = TailCall$.class, id = 1208) +@Persistable(clazz = TypeNames$.class, id = 1209) +@Persistable(clazz = TypeSignatures$.class, id = 1210) +@Persistable(clazz = DocumentationComments$.class, id = 1211) +@Persistable(clazz = ModuleAnnotations$.class, id = 1212) +@Persistable(clazz = GatherDiagnostics$.class, id = 1213) +@Persistable(clazz = MethodCalls$.class, id = 1214) +@Persistable(clazz = MethodDefinitions$.class, id = 1215) +@Persistable(clazz = GenericAnnotations$.class, id = 1216) +@Persistable(clazz = ExpressionAnnotations$.class, id = 1217) +@Persistable(clazz = FullyQualifiedNames$.class, id = 1218) public final class PassPersistance { private PassPersistance() {} diff --git a/engine/runtime-compiler/src/main/scala/org/enso/compiler/phase/ImportResolver.scala b/engine/runtime-compiler/src/main/scala/org/enso/compiler/phase/ImportResolver.scala index 12b061d2a56d..9ca7d62a3dda 100644 --- a/engine/runtime-compiler/src/main/scala/org/enso/compiler/phase/ImportResolver.scala +++ b/engine/runtime-compiler/src/main/scala/org/enso/compiler/phase/ImportResolver.scala @@ -20,6 +20,7 @@ import org.enso.compiler.pass.analyse.BindingAnalysis import org.enso.editions.LibraryName import org.enso.polyglot.CompilationStage import scala.collection.mutable +import java.io.IOException /** Runs imports resolution. Starts from a given module and then recursively * collects all modules that are reachable from it. @@ -47,11 +48,27 @@ class ImportResolver(compiler: Compiler) { def analyzeModule(current: Module): List[Module] = { val context = compiler.context - val ir = context.getIr(current) - val currentLocal = ir.unsafeGetMetadata( - BindingAnalysis, - "Non-parsed module used in ImportResolver" - ) + val (ir, currentLocal) = + try { + val ir = context.getIr(current) + val currentLocal = ir.unsafeGetMetadata( + BindingAnalysis, + "Non-parsed module used in ImportResolver" + ) + (ir, currentLocal) + } catch { + case _: IOException => + context.updateModule( + current, + u => { + u.ir(null) + u.compilationStage(CompilationStage.INITIAL) + u.invalidateCache() + } + ) + compiler.ensureParsed(current) + return analyzeModule(current) + } // put the list of resolved imports in the module metadata if ( context @@ -156,9 +173,21 @@ class ImportResolver(compiler: Compiler) { val mod = name.parts.dropRight(1).map(_.name).mkString(".") compiler.getModule(mod).flatMap { mod => compiler.ensureParsed(mod) - mod - .getBindingsMap() - .definedEntities + var b = mod.getBindingsMap() + if (b == null) { + compiler.context.updateModule( + mod, + { u => + u.invalidateCache() + u.ir(null) + u.compilationStage(CompilationStage.INITIAL) + } + ) + compiler.ensureParsed(mod) + b = mod.getBindingsMap() + } + + b.definedEntities .find(_.name == tp) .collect { case t: Type => ResolvedType(ModuleReference.Concrete(mod), t) diff --git a/engine/runtime-parser/src/main/java/org/enso/compiler/core/ir/IrLazyMap.java b/engine/runtime-parser/src/main/java/org/enso/compiler/core/ir/IrLazyMap.java new file mode 100644 index 000000000000..d6f554f4cede --- /dev/null +++ b/engine/runtime-parser/src/main/java/org/enso/compiler/core/ir/IrLazyMap.java @@ -0,0 +1,85 @@ +package org.enso.compiler.core.ir; + +import java.io.IOException; +import java.util.AbstractMap; +import java.util.LinkedHashMap; +import java.util.LinkedHashSet; +import java.util.Map; +import java.util.Objects; +import java.util.Set; + +import org.enso.persist.Persistance; +import org.enso.persist.Persistance.Reference; + + +final class IrLazyMap extends AbstractMap { + private final Map> delegate; + + @SuppressWarnings("unchecked") + IrLazyMap(Persistance.Input in) throws IOException { + var map = new LinkedHashMap>(); + var n = in.readInt(); + for (var i = 0; i < n; i++) { + var key = (K) in.readObject(); + var ref = (Reference) in.readReference(Object.class); + var en = new En<>(key, ref); + map.put(key, en); + } + this.delegate = map; + } + + @Override + public Set> entrySet() { + return new LinkedHashSet<>(this.delegate.values()); + } + + @Override + public V get(Object key) { + var entry = this.delegate.get(key); + return entry == null ? null : entry.getValue(); + } + + private static final class En implements Entry { + private final K key; + private final Reference ref; + + En(K key, Reference ref) { + this.key = key; + this.ref = ref; + } + + @Override + public K getKey() { + return key; + } + + @Override + @SuppressWarnings("unchecked") + public V getValue() { + return (V) ref.get(Object.class); + } + + @Override + public V setValue(V value) { + throw new UnsupportedOperationException(); + } + + @Override + public int hashCode() { + int hash = 7; + hash = 29 * hash + Objects.hashCode(this.key); + return hash; + } + + @Override + public boolean equals(Object obj) { + if (this == obj) { + return true; + } + if (obj instanceof En other) { + return Objects.equals(this.key, other.key); + } + return false; + } + } +} diff --git a/engine/runtime-parser/src/main/java/org/enso/compiler/core/ir/IrPersistance.java b/engine/runtime-parser/src/main/java/org/enso/compiler/core/ir/IrPersistance.java index 4a7a986ac8ac..349bf53a6ff3 100644 --- a/engine/runtime-parser/src/main/java/org/enso/compiler/core/ir/IrPersistance.java +++ b/engine/runtime-parser/src/main/java/org/enso/compiler/core/ir/IrPersistance.java @@ -1,7 +1,6 @@ package org.enso.compiler.core.ir; import java.io.IOException; -import java.util.HashMap; import java.util.Map; import java.util.UUID; import org.enso.compiler.core.ir.expression.Application; @@ -230,14 +229,9 @@ protected void writeObject(scala.collection.immutable.Map map, Output out) throw @SuppressWarnings("unchecked") protected scala.collection.immutable.Map readObject(Input in) throws IOException, ClassNotFoundException { - var size = in.readInt(); - var map = scala.collection.immutable.Map$.MODULE$.empty(); - for (var i = 0; i < size; i++) { - var key = in.readObject(); - var value = in.readObject(); - map = map.$plus(new Tuple2(key, value)); - } - return map; + var map = new IrLazyMap(in); + var immutableMap = new IrLazyImMap(map); + return immutableMap; } } @@ -309,13 +303,13 @@ protected scala.collection.immutable.Set readObject(Input in) } @ServiceProvider(service = Persistance.class) - public static final class PersistMap extends Persistance { + public static final class PersistMap extends Persistance { public PersistMap() { - super(HashMap.class, true, 4440); + super(Map.class, true, 4440); } @Override - protected void writeObject(HashMap m, Output out) throws IOException { + protected void writeObject(Map m, Output out) throws IOException { var size = m.size(); out.writeInt(size); var it = m.entrySet().iterator(); @@ -328,15 +322,8 @@ protected void writeObject(HashMap m, Output out) throws IOException { @Override @SuppressWarnings("unchecked") - protected HashMap readObject(Input in) throws IOException, ClassNotFoundException { - var size = in.readInt(); - var map = new HashMap(); - for (var i = 0; i < size; i++) { - var key = in.readObject(); - var value = in.readObject(); - map.put(key, value); - } - return map; + protected Map readObject(Input in) throws IOException, ClassNotFoundException { + return new IrLazyMap(in); } } @@ -370,7 +357,7 @@ protected Seq readObject(Input in) throws IOException, ClassNotFoundException { @ServiceProvider(service = Persistance.class) public static final class PersistMetadataStorage extends Persistance { public PersistMetadataStorage() { - super(MetadataStorage.class, false, 301); + super(MetadataStorage.class, false, 381); } @Override @@ -379,7 +366,7 @@ protected void writeObject(MetadataStorage obj, Output out) throws IOException { var map = obj.map( (processingPass, data) -> { - var t = new Tuple2<>(processingPass.getClass().getName(), data); + var t = new Tuple2<>(processingPass, data); return t; }); out.writeInline(scala.collection.immutable.Map.class, map); @@ -388,19 +375,8 @@ protected void writeObject(MetadataStorage obj, Output out) throws IOException { @Override @SuppressWarnings("unchecked") protected MetadataStorage readObject(Input in) throws IOException, ClassNotFoundException { - var storage = new MetadataStorage(nil()); var map = in.readInline(scala.collection.immutable.Map.class); - var it = map.iterator(); - while (it.hasNext()) { - var obj = (Tuple2) it.next(); - try { - var pass = (ProcessingPass) Class.forName(obj._1()).getField("MODULE$").get(null); - var data = obj._2(); - storage.update(pass, data); - } catch (ReflectiveOperationException ex) { - throw new IOException(ex); - } - } + var storage = new MetadataStorage(map); return storage; } } diff --git a/engine/runtime-parser/src/main/scala/org/enso/compiler/core/ir/IrLazyImMap.scala b/engine/runtime-parser/src/main/scala/org/enso/compiler/core/ir/IrLazyImMap.scala new file mode 100644 index 000000000000..9a64641a34c5 --- /dev/null +++ b/engine/runtime-parser/src/main/scala/org/enso/compiler/core/ir/IrLazyImMap.scala @@ -0,0 +1,37 @@ +package org.enso.compiler.core.ir + +import scala.collection.immutable.StrictOptimizedMapOps +import scala.collection.immutable.Map +import scala.collection.Iterator + +final private class IrLazyImMap[K, V]( + val underlying: java.util.Map[K, V] +) extends Map[K, V] + with StrictOptimizedMapOps[K, V, Map, Map[K, V]] { + def removed(key: K) = + new scala.jdk.CollectionConverters.MapHasAsScala(underlying).asScala.toMap + .removed(key) + + def updated[V1 >: V](key: K, value: V1) = + new scala.jdk.CollectionConverters.MapHasAsScala(underlying).asScala.toMap + .updated(key, value) + + def get(key: K) = Option(underlying.get(key)) + + def iterator = new Iterator[(K, V)]() { + val keys = underlying.entrySet().iterator() + def hasNext: Boolean = keys.hasNext() + def next(): (K, V) = { + val entry = keys.next() + (entry.getKey, entry.getValue) + } + } + + override def size = underlying.size + + // Has to override the `contains` method because the underlying implementation + // calls `get(key)` + override def contains(key: K): Boolean = { + underlying.containsKey(key) + } +} diff --git a/engine/runtime-parser/src/main/scala/org/enso/compiler/core/ir/MetadataStorage.scala b/engine/runtime-parser/src/main/scala/org/enso/compiler/core/ir/MetadataStorage.scala index 7a849b780876..207fe21e9744 100644 --- a/engine/runtime-parser/src/main/scala/org/enso/compiler/core/ir/MetadataStorage.scala +++ b/engine/runtime-parser/src/main/scala/org/enso/compiler/core/ir/MetadataStorage.scala @@ -12,11 +12,15 @@ import org.enso.compiler.core.CompilerStub */ //noinspection DuplicatedCode final class MetadataStorage( - startingMeta: Seq[MetadataPair[_]] = Seq() -) extends Serializable { - private var metadata: Map[ProcessingPass, Any] = Map( - startingMeta.map(_.asPair.asInstanceOf[(ProcessingPass, Any)]): _* - ) + private var metadata: Map[ProcessingPass, Any] +) { + def this(startingMeta: Seq[MetadataPair[_]] = Seq()) = { + this( + Map( + startingMeta.map(_.asPair.asInstanceOf[(ProcessingPass, Any)]): _* + ) + ) + } /** Adds a metadata pair to the node metadata. * diff --git a/engine/runtime-parser/src/test/java/org/enso/compiler/core/IrPersistanceTest.java b/engine/runtime-parser/src/test/java/org/enso/compiler/core/IrPersistanceTest.java index 9ac7e4a9d8f1..cb0142e9aa3d 100644 --- a/engine/runtime-parser/src/test/java/org/enso/compiler/core/IrPersistanceTest.java +++ b/engine/runtime-parser/src/test/java/org/enso/compiler/core/IrPersistanceTest.java @@ -16,6 +16,7 @@ import org.enso.compiler.core.ir.Module; import org.enso.persist.Persistable; import org.enso.persist.Persistance; +import org.junit.Before; import org.junit.Test; import org.openide.util.lookup.ServiceProvider; @@ -25,6 +26,11 @@ import scala.collection.immutable.Seq; public class IrPersistanceTest { + @Before + public void resetDebris() { + LazySeq.forbidden = false; + } + @Test public void locationTest() throws Exception { var l = new Location(12, 33); @@ -61,6 +67,30 @@ public void scalaMap() throws Exception { assertEquals(in, out); } + @Test + @SuppressWarnings("unchecked") + public void scalaImmutableMapIsLazy() throws Exception { + var s1 = new LazySeq("Hello"); + var s2 = new LazySeq("World"); + var in = (scala.collection.immutable.Map) scala.collection.immutable.Map$.MODULE$.empty() + .$plus(new Tuple2("Hello", s1)) + .$plus(new Tuple2("World", s2)); + + LazySeq.forbidden = true; + var out = (scala.collection.immutable.Map) + serde(scala.collection.immutable.Map.class, in, 64); + + assertEquals("Two pairs element", 2, out.size()); + assertEquals("Two keys", 2, out.keySet().size()); + assertTrue(out.keySet().contains("Hello")); + assertTrue(out.keySet().contains("World")); + LazySeq.forbidden = false; + + assertEquals(s1, out.get("Hello").get()); + assertEquals(s2, out.get("World").get()); + assertEquals(in, out); + } + @Test @SuppressWarnings("unchecked") public void scalaHashMap() throws Exception { @@ -184,11 +214,34 @@ public void hashMap() throws Exception { map.put("two", "duo"); map.put("ten", "tre"); - var out = serde(HashMap.class, map, -1); + var out = serde(java.util.Map.class, map, -1); assertEquals("Same", map, out); } + @Test + @SuppressWarnings("unchecked") + public void hashMapIsLazy() throws Exception { + var s1 = new LazySeq("Hello"); + var s2 = new LazySeq("World"); + var in = new HashMap(); + in.put("Hello", s1); + in.put("World", s2); + + LazySeq.forbidden = true; + var out = serde(java.util.Map.class, in, 64); + + assertEquals("Two pairs element", 2, out.size()); + assertEquals("Two keys", 2, out.keySet().size()); + assertTrue(out.keySet().contains("Hello")); + assertTrue(out.keySet().contains("World")); + LazySeq.forbidden = false; + + assertEquals(s1, out.get("Hello")); + assertEquals(s2, out.get("World")); + assertEquals(in, out); + } + @Test public void readResolve() throws Exception { var in = new Service(5); diff --git a/engine/runtime/src/main/java/org/enso/interpreter/runtime/TruffleCompilerContext.java b/engine/runtime/src/main/java/org/enso/interpreter/runtime/TruffleCompilerContext.java index cecfa8b21eb9..23d28d9b081f 100644 --- a/engine/runtime/src/main/java/org/enso/interpreter/runtime/TruffleCompilerContext.java +++ b/engine/runtime/src/main/java/org/enso/interpreter/runtime/TruffleCompilerContext.java @@ -1,14 +1,5 @@ package org.enso.interpreter.runtime; -import org.enso.compiler.Passes; -import org.enso.compiler.pass.analyse.BindingAnalysis$; -import org.enso.compiler.context.CompilerContext; -import org.enso.compiler.context.FreshNameSupply; - -import com.oracle.truffle.api.TruffleLogger; -import com.oracle.truffle.api.TruffleFile; -import com.oracle.truffle.api.source.Source; - import java.io.IOException; import java.io.PrintStream; import java.util.List; @@ -20,8 +11,12 @@ import org.enso.compiler.Compiler; import org.enso.compiler.PackageRepository; +import org.enso.compiler.Passes; +import org.enso.compiler.context.CompilerContext; +import org.enso.compiler.context.FreshNameSupply; import org.enso.compiler.data.BindingsMap; import org.enso.compiler.data.CompilerConfig; +import org.enso.compiler.pass.analyse.BindingAnalysis$; import org.enso.editions.LibraryName; import org.enso.interpreter.caches.Cache; import org.enso.interpreter.caches.ModuleCache; @@ -29,8 +24,13 @@ import org.enso.pkg.Package; import org.enso.pkg.QualifiedName; import org.enso.polyglot.CompilationStage; +import org.enso.polyglot.LanguageInfo; import org.enso.polyglot.data.TypeGraph; +import com.oracle.truffle.api.TruffleFile; +import com.oracle.truffle.api.TruffleLogger; +import com.oracle.truffle.api.source.Source; + import scala.Option; final class TruffleCompilerContext implements CompilerContext { @@ -277,7 +277,7 @@ public void shutdown(boolean waitForPendingJobCompletion) { private final class ModuleUpdater implements Updater, AutoCloseable { private final Module module; private BindingsMap map; - private org.enso.compiler.core.ir.Module ir; + private org.enso.compiler.core.ir.Module[] ir; private CompilationStage stage; private Boolean loadedFromCache; private boolean resetScope; @@ -294,7 +294,7 @@ public void bindingsMap(BindingsMap map) { @Override public void ir(org.enso.compiler.core.ir.Module ir) { - this.ir = ir; + this.ir = new org.enso.compiler.core.ir.Module[] { ir }; } @Override @@ -326,7 +326,7 @@ public void close() { module.bindings = map; } if (ir != null) { - module.module.unsafeSetIr(ir); + module.module.unsafeSetIr(ir[0]); } if (stage != null) { module.module.unsafeSetCompilationStage(stage); @@ -380,12 +380,19 @@ public QualifiedName getName() { @Override public BindingsMap getBindingsMap() { if (module.getIr() != null) { - var meta = module.getIr().passData(); - var pass = meta.get(BindingAnalysis$.MODULE$); - return (BindingsMap) pass.get(); - } else { - return bindings; + try { + var meta = module.getIr().passData(); + var pass = meta.get(BindingAnalysis$.MODULE$); + emitIOException(); + return (BindingsMap) pass.get(); + } catch (IOException ex) { + var logger = TruffleLogger.getLogger(LanguageInfo.ID, org.enso.interpreter.runtime.Module.class); + var msg = "Cannot read BindingsMap for " + getName() + ": " + ex.getMessage(); + logger.log(Level.SEVERE, msg); + logger.log(Level.FINE, msg, ex); + } } + return bindings; } @Override @@ -453,4 +460,7 @@ public String toString() { return sb.toString(); } } + + private static void emitIOException() throws IOException { + } } diff --git a/lib/java/persistance-dsl/src/main/java/org/enso/persist/impl/PersistableProcessor.java b/lib/java/persistance-dsl/src/main/java/org/enso/persist/impl/PersistableProcessor.java index 16076d46d61b..4cc6f913bf37 100644 --- a/lib/java/persistance-dsl/src/main/java/org/enso/persist/impl/PersistableProcessor.java +++ b/lib/java/persistance-dsl/src/main/java/org/enso/persist/impl/PersistableProcessor.java @@ -1,6 +1,7 @@ package org.enso.persist.impl; import java.io.IOException; +import java.lang.module.ModuleDescriptor; import java.util.ArrayList; import java.util.Comparator; import java.util.List; @@ -115,17 +116,31 @@ public int compare(Object a, Object b) { ) .sorted(richerConstructor) .toList(); + + ExecutableElement cons; + Element singleton; if (constructors.isEmpty()) { - processingEnv.getMessager().printMessage(Kind.ERROR, "There should be exactly one constructor in " + typeElem, orig); - return false; - } - var cons = (ExecutableElement) constructors.get(0); - if (constructors.size() > 1) { - var snd = (ExecutableElement) constructors.get(1); - if (richerConstructor.compare(cons, snd) == 0) { - processingEnv.getMessager().printMessage(Kind.ERROR, "There should be exactly one 'richest' constructor in " + typeElem, orig); + var singletonFields = typeElem.getEnclosedElements().stream().filter( + e -> e.getKind() == ElementKind.FIELD && e.getModifiers().contains(Modifier.STATIC) && e.getModifiers().contains(Modifier.PUBLIC) + ).filter( + e -> tu.isSameType(e.asType(), typeElem.asType()) + ).toList(); + if (singletonFields.isEmpty()) { + processingEnv.getMessager().printMessage(Kind.ERROR, "There should be exactly one constructor in " + typeElem, orig); return false; } + singleton = singletonFields.get(0); + cons = null; + } else { + cons = (ExecutableElement) constructors.get(0); + singleton = null; + if (constructors.size() > 1) { + var snd = (ExecutableElement) constructors.get(1); + if (richerConstructor.compare(cons, snd) == 0) { + processingEnv.getMessager().printMessage(Kind.ERROR, "There should be exactly one 'richest' constructor in " + typeElem, orig); + return false; + } + } } var pkgName = eu.getPackageOf(orig).getQualifiedName().toString(); var className = "Persist" + findNameInPackage(typeElem).replace(".", "_"); @@ -137,68 +152,75 @@ public int compare(Object a, Object b) { w.append("@org.openide.util.lookup.ServiceProvider(service=Persistance.class)\n"); w.append("public final class ").append(className).append(" extends Persistance<").append(typeElemName).append("> {\n"); w.append(" public ").append(className).append("() {\n"); - var id = readAnnoValue(anno, "id").toString(); // Integer.toString(anno.id()); + var id = readAnnoValue(anno, "id"); w.append(" super(").append(typeElemName).append(".class, false, ").append(id).append(");\n"); w.append(" }\n"); w.append(" @SuppressWarnings(\"unchecked\")\n"); + w.append(" @Override\n"); w.append(" protected ").append(typeElemName).append(" readObject(Input in) throws IOException {\n"); - for (var v : cons.getParameters()) { - if (tu.isSameType(eu.getTypeElement("java.lang.String").asType(), v.asType())) { - w.append(" var ").append(v.getSimpleName()).append(" = in.readUTF();\n"); - } else if (!v.asType().getKind().isPrimitive()) { - var type = tu.erasure(v.asType()); - var elem = (TypeElement) tu.asElement(type); - var name = findFqn(elem); - if (shouldInline(elem)) { - w.append(" var ").append(v.getSimpleName()).append(" = in.readInline(").append(name).append(".class);\n"); - } else { - w.append(" var ").append(v.getSimpleName()).append(" = (").append(name).append(") in.readObject();\n"); + if (cons != null) { + for (var v : cons.getParameters()) { + if (tu.isSameType(eu.getTypeElement("java.lang.String").asType(), v.asType())) { + w.append(" var ").append(v.getSimpleName()).append(" = in.readUTF();\n"); + } else if (!v.asType().getKind().isPrimitive()) { + var type = tu.erasure(v.asType()); + var elem = (TypeElement) tu.asElement(type); + var name = findFqn(elem); + if (shouldInline(elem)) { + w.append(" var ").append(v.getSimpleName()).append(" = in.readInline(").append(name).append(".class);\n"); + } else { + w.append(" var ").append(v.getSimpleName()).append(" = (").append(name).append(") in.readObject();\n"); + } + } else switch (v.asType().getKind()) { + case BOOLEAN -> + w.append(" var ").append(v.getSimpleName()).append(" = in.readBoolean();\n"); + case INT -> + w.append(" var ").append(v.getSimpleName()).append(" = in.readInt();\n"); + default -> + processingEnv.getMessager().printMessage(Kind.ERROR, "Unsupported primitive type: " + v.asType().getKind()); } - } else switch (v.asType().getKind()) { - case BOOLEAN -> - w.append(" var ").append(v.getSimpleName()).append(" = in.readBoolean();\n"); - case INT -> - w.append(" var ").append(v.getSimpleName()).append(" = in.readInt();\n"); - default -> - processingEnv.getMessager().printMessage(Kind.ERROR, "Unsupported primitive type: " + v.asType().getKind()); } - } - w.append(" return new ").append(typeElemName).append("(\n"); - w.append(" "); - { - var sep = ""; - for (var v : cons.getParameters()) { - w.append(sep); - w.append(v.getSimpleName()); - sep = ", "; + w.append(" return new ").append(typeElemName).append("(\n"); + w.append(" "); + { + var sep = ""; + for (var v : cons.getParameters()) { + w.append(sep); + w.append(v.getSimpleName()); + sep = ", "; + } } + w.append("\n"); + w.append(" );\n"); + } else { + w.append(" return ").append(typeElemName).append(".").append(singleton.getSimpleName()).append(";\n"); } - w.append("\n"); - w.append(" );\n"); w.append(" }\n"); w.append(" @SuppressWarnings(\"unchecked\")\n"); + w.append(" @Override\n"); w.append(" protected void writeObject(").append(typeElemName).append(" obj, Output out) throws IOException {\n"); - - for (var v : cons.getParameters()) { - if (tu.isSameType(eu.getTypeElement("java.lang.String").asType(), v.asType())) { - w.append(" out.writeUTF(obj.").append(v.getSimpleName()).append("());\n"); - } else if (!v.asType().getKind().isPrimitive()) { - var type = tu.erasure(v.asType()); - var elem = (TypeElement) tu.asElement(type); - var name = findFqn(elem); - if (shouldInline(elem)) { - w.append(" out.writeInline(").append(name).append(".class, obj.").append(v.getSimpleName()).append("());\n"); - } else { - w.append(" out.writeObject(obj.").append(v.getSimpleName()).append("());\n"); + if (cons != null) { + for (var v : cons.getParameters()) { + if (tu.isSameType(eu.getTypeElement("java.lang.String").asType(), v.asType())) { + w.append(" out.writeUTF(obj.").append(v.getSimpleName()).append("());\n"); + } else if (!v.asType().getKind().isPrimitive()) { + var type = tu.erasure(v.asType()); + var elem = (TypeElement) tu.asElement(type); + var name = findFqn(elem); + if (shouldInline(elem)) { + w.append(" out.writeInline(").append(name).append(".class, obj.").append(v.getSimpleName()).append("());\n"); + } else { + w.append(" out.writeObject(obj.").append(v.getSimpleName()).append("());\n"); + } + } else switch (v.asType().getKind()) { + case BOOLEAN -> + w.append(" out.writeBoolean(obj.").append(v.getSimpleName()).append("());\n"); + case INT -> + w.append(" out.writeInt(obj.").append(v.getSimpleName()).append("());\n"); + default -> + processingEnv.getMessager().printMessage(Kind.ERROR, "Unsupported primitive type: " + v.asType().getKind()); } - } else switch (v.asType().getKind()) { - case BOOLEAN -> - w.append(" out.writeBoolean(obj.").append(v.getSimpleName()).append("());\n"); - case INT -> - w.append(" out.writeInt(obj.").append(v.getSimpleName()).append("());\n"); - default -> - processingEnv.getMessager().printMessage(Kind.ERROR, "Unsupported primitive type: " + v.asType().getKind()); } } w.append(" }\n"); diff --git a/lib/java/persistance/src/main/java/org/enso/persist/PerInputImpl.java b/lib/java/persistance/src/main/java/org/enso/persist/PerInputImpl.java index d26f0ff256b6..1517df297acf 100644 --- a/lib/java/persistance/src/main/java/org/enso/persist/PerInputImpl.java +++ b/lib/java/persistance/src/main/java/org/enso/persist/PerInputImpl.java @@ -31,7 +31,7 @@ static Reference readObject(byte[] arr, Function readReso var buf = ByteBuffer.wrap(arr); var version = buf.getInt(4); if (version != PerMap.DEFAULT.versionStamp) { - throw new IOException("Incompatible version " + version + " != " + PerMap.DEFAULT.versionStamp); + throw new IOException("Incompatible version " + Integer.toHexString(version) + " != " + Integer.toHexString(PerMap.DEFAULT.versionStamp)); } var at = buf.getInt(8); var cache = new InputCache(buf, readResolve); diff --git a/test/Benchmarks/src/Main.enso b/test/Benchmarks/src/Main.enso index 5823d8acc195..1e18ab43cbb1 100644 --- a/test/Benchmarks/src/Main.enso +++ b/test/Benchmarks/src/Main.enso @@ -30,7 +30,7 @@ import project.Number_Parse import project.Numeric import project.Range import project.Sum -import project.Startup +import project.Startup.Startup import project.Runtime.Panics_And_Errors from Standard.Base.Runtime import Debug diff --git a/test/Benchmarks/src/Startup.enso b/test/Benchmarks/src/Startup.enso deleted file mode 100644 index 3d1901f8145c..000000000000 --- a/test/Benchmarks/src/Startup.enso +++ /dev/null @@ -1,52 +0,0 @@ -from Standard.Base import all - -from Standard.Test import Bench - -polyglot java import java.lang.System as Java_System -polyglot java import java.io.File as Java_File - -type Data - Value ~enso_bin:File ~empty_world:File ~hello_world:File - - bench_empty self = - res = Process.run self.enso_bin.path [ "--run", self.empty_world.to_text ] - - bench_hello self = - res = Process.run self.enso_bin.path [ "--run", self.hello_world.to_text ] - - -collect_benches = Bench.build builder-> - options = Bench.options . set_warmup (Bench.phase_conf 2 5) . set_measure (Bench.phase_conf 3 5) - - - data = - Data.Value enso_bin empty_file hello_file - - builder.group "Startup" options group_builder-> - group_builder.specify "empty_startup" data.bench_empty - group_builder.specify "hello_world_startup" data.bench_hello - -empty_file = - f = File.create_temporary_file "empty" "enso" - t = """ - main = "Hello World" - t.write f - f - -hello_file = - f = File.create_temporary_file "hello" "enso" - t = """ - from Standard.Base import IO - - main = IO.println "Hello World" - t.write f - f - -enso_bin = - p = Java_System.getProperty "jdk.module.path" - s = p.split Java_File.separator - paths = s.take (Index_Sub_Range.While _!="..") - j = paths . join Java_File.separator - File.new j / if Platform.os == Platform.OS.Windows then "enso.bat" else "enso" - -main = collect_benches . run_main diff --git a/test/Benchmarks/src/Startup/Empty_World.enso b/test/Benchmarks/src/Startup/Empty_World.enso new file mode 100644 index 000000000000..6e147c44da22 --- /dev/null +++ b/test/Benchmarks/src/Startup/Empty_World.enso @@ -0,0 +1 @@ +main = "Hello World" diff --git a/test/Benchmarks/src/Startup/Hello_World.enso b/test/Benchmarks/src/Startup/Hello_World.enso new file mode 100644 index 000000000000..4e427a9e38f7 --- /dev/null +++ b/test/Benchmarks/src/Startup/Hello_World.enso @@ -0,0 +1,3 @@ +from Standard.Base import IO + +main = IO.println "Hello World" diff --git a/test/Benchmarks/src/Startup/Startup.enso b/test/Benchmarks/src/Startup/Startup.enso new file mode 100644 index 000000000000..51703c32a6e4 --- /dev/null +++ b/test/Benchmarks/src/Startup/Startup.enso @@ -0,0 +1,70 @@ +from Standard.Base import all + +from Standard.Test import Bench + +polyglot java import java.lang.System as Java_System +polyglot java import java.io.File as Java_File + +type Data + Value ~enso_bin:File ~empty_world:File ~hello_world:File + + bench_empty self = + self.startup [ "--run", self.empty_world.to_text ] + + bench_hello self = + self.startup [ "--run", self.hello_world.to_text ] + + startup self args = + exe = self.enso_bin + result = Process.run exe.path args + case result.exit_code of + Exit_Code.Failure code -> + IO.println "Exit code: "+code.to_text + IO.println result.stdout + IO.println result.stderr + Panic.throw "Exit code: "+code.to_text + Exit_Code.Success -> + if result.stdout.contains "Hello World" then Nothing else + msg = "Execution should contain 'Hello World', but:\n====\n" + result.stdout + '\n' + result.stderr + "\n====" + IO.println msg + Panic.throw msg + +collect_benches = Bench.build builder-> + options = Bench.options . set_warmup (Bench.phase_conf 2 5) . set_measure (Bench.phase_conf 3 5) + + + data = + Data.Value enso_bin (find_sibling "Empty_World.enso") (find_sibling "Hello_World.enso") + + builder.group "Startup" options group_builder-> + group_builder.specify "empty_startup" data.bench_empty + group_builder.specify "hello_world_startup" data.bench_hello + +find_sibling name = + f = enso_project.root / "src" / "Startup" / name + if f.is_regular_file.not then Panic.throw "Cannot find "+f.to_text + f + +enso_bin = + find_prefix dir prefix = + vec = dir.list name_filter=prefix+"*" + if vec.length == 1 then vec.at 0 else + msg = "Cannot find " + prefix + "* in" + dir.to_text + "\n" + err = dir.list.fold msg t-> f-> + t + f.to_text + "\n" + Panic.throw err + + project_root = File.new enso_project.root.to_text + repository_root = project_root . parent . parent + built_distribution = find_prefix repository_root "built-distribution" + enso_engine = find_prefix built_distribution "enso-engine-" + enso = find_prefix enso_engine "enso-" + bin = find_prefix enso "bin" + + exe = File.new bin / if Platform.os == Platform.OS.Windows then "enso.bat" else "enso" + + if exe.is_regular_file.not then Panic.throw "Cannot find "+exe.to_text + + exe + +main = collect_benches . run_main