Skip to content

Commit

Permalink
Merging with lazy MetadataStorage
Browse files Browse the repository at this point in the history
  • Loading branch information
JaroslavTulach committed Nov 27, 2023
2 parents ef9b0a5 + 893965e commit 48f9651
Show file tree
Hide file tree
Showing 19 changed files with 539 additions and 235 deletions.
4 changes: 1 addition & 3 deletions docs/debugger/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,5 @@ used by Enso, broken up as follows:
Enso code using Chrome devtools.
- [**Debug adapter protocol:**](./dap.md) A guide how to debug Enso code using
VSCode.
- [**Debugging Enso and Java code at once:**](./mixed-debugging.md) A
step-by-step guide how to debug both Enso and Java code in a single debugger.
- [**Debugging Engine (Runtime) only Java code:**](./runtime-debugging.md) A
guide how to debug the internal Engine Java code.
guide how to debug Enso code & the internal Java code.
47 changes: 1 addition & 46 deletions docs/debugger/mixed-debugging.md
Original file line number Diff line number Diff line change
@@ -1,48 +1,3 @@
# Debugging Enso and Java Code at Once

Enso libraries are written in a mixture of Enso code and Java libraries.
Debugging both sides (the Java as well as Enso code) is possible with a decent
IDE.

Get [NetBeans](http://netbeans.apache.org) version 13 or newer or
[VS Code with Apache Language Server extension](https://cwiki.apache.org/confluence/display/NETBEANS/Apache+NetBeans+Extension+for+Visual+Studio+Code)
and _start listening on port 5005_ with _Debug/Attach Debugger_ or by specifying
following debug configuration in VSCode:

```json
{
"name": "Listen to 5005",
"type": "java+",
"request": "attach",
"listen": "true",
"hostName": "localhost",
"port": "5005"
}
```

Then it is just about executing following Sbt command which builds CLI version
of the engine, launches it in debug mode and passes all other arguments to the
started process:

```bash
sbt:enso> runEngineDistribution --debug --run ./test/Tests/src/Data/Numbers_Spec.enso
```

Alternatively you can pass in special JVM arguments when launching the
`bin/enso` launcher:

```bash
enso$ JAVA_OPTS=-agentlib:jdwp=transport=dt_socket,server=n,address=5005 ./built-distribution/enso-engine-*/enso-*/bin/enso --run ./test/Tests/src/Data/Numbers_Spec.enso
```

As soon as the debuggee connects and the Enso language starts - choose the
_Toggle Pause in GraalVM Script_ button in the toolbar:

![NetBeans Debugger](https://user-images.githubusercontent.com/26887752/209614191-b0513635-819b-4c64-a6f9-9823b90a1513.png)

and your execution shall stop on the next `.enso` line of code. This mode allows
to debug both - the Enso code as well as Java code. The stack traces shows a
mixture of Java and Enso stack frames by default. Right-clicking on the thread
allows one to switch to plain Java view (with a way more stack frames) and back.
Analyzing low level details as well as Enso developer point of view shall be
simple with such tool.
Desribed at [Runtime Debugging](runtime-debugging.md).
116 changes: 98 additions & 18 deletions docs/debugger/runtime-debugging.md
Original file line number Diff line number Diff line change
@@ -1,37 +1,117 @@
# Runtime (Engine) debugging
# High & Low Runtime Debugging

This section explains how to debug various parts of the Engine. By Engine, we
mean all the Java code located in the `runtime` SBT project, in `engine`
directory.
This document describes how to debug **everything** - e.g. how to debug the
_Java code_ that powers the Enso _engine & interpreter_ as well as Enso standard
_libraries Java code_ (used to interact with the operating system via Java
APIs). At the same thetime the document describes how to _map that information_
onto the execution of the actual _Enso source code_ being interpreted.

## Debugging source file evaluation
## Debugging Single Source File

This subsection provides a guide how to debug a single Enso source file
evaluation. To evaluate a single source file, we use the _Engine distribution_
built with `buildEngineDistribution` command. Both of the following two options
starts the JVM in a debug mode. After the JVM is started in a debug mode, simply
attach the debugger to the JVM process at the specified port.
To analyze what is happening in the interpreter, we often want to debug
excecution of a _single Enso file_.

The first option is to invoke the engine distribution from SBT shell with:
Get started by building the _Engine distribution_ built with
`sbt buildEngineDistribution` command. Then configure your IDE to understand the
_engine Java & Scala sources_ which are mostly located in the
[engine subdirectory](https://github.com/enso-org/enso/tree/develop/engine). The
sources are known to be understandable by:

- VSCode - with
[Enso Extension described herein](../../tools/enso4igv/README.md)
- IntelliJ
- IGV - with [extension described herein](../../tools/enso4igv/IGV.md)
- [NetBeans](http://netbeans.apache.org) - with the same
[extension as IGV](../../tools/enso4igv/IGV.md) is using

There are two ways to start JVM in a debug mode. The first one is fully
integrated into the `sbt` build system. It builds the engine (in case there were
any changes) and then starts the JVM in debug mode trying to attach to port
`5005`:

```sh
sbt:enso> runEngineDistribution --debug --run ./test/Tests/src/Data/Numbers_Spec.enso
```

The second options is to pass in special JVM arguments when launching the
`bin/enso` from the engine distribution:
The second options gives one a complete control as it launches everything from a
command line. By specifying `JAVA_OPTS` environment variable one influences the
special JVM arguments when launching the `bin/enso` from the engine
distribution:

```bash
enso$ JAVA_OPTS=-agentlib:jdwp=transport=dt_socket,server=n,address=5005 ./built-distribution/enso-engine-*/enso-*/bin/enso --run ./test/Tests/src/Data/Numbers_Spec.enso
```

### Tips and tricks
Both of the approaches launch the JVM in a _debug mode_. Once the JVM is
started, simply attach the debugger to the JVM process at the specified port
(usually `5005` is used).

### Attaching from VSCode

First of all make sure your VSCode contains the
[Enso supporting extension](https://marketplace.visualstudio.com/items?itemName=Enso.enso4vscode).
Follow the [instructions provided herein](../../tools/enso4igv/README.md) to
install everything that's needed.

Once the
[Enso extension](https://marketplace.visualstudio.com/items?itemName=Enso.enso4vscode)
is installed, open the root of Enso repository as workspace and select _"Listen
to 5005"_ debug configuration:

![Listen to 5005](https://github.com/enso-org/enso/assets/26887752/1874bcb1-cf8b-4df4-92d8-e7fb57e1b17a)

Once the connection with the Enso JVM is made, debug the Java code as usual.

### Mapping Java and Enso Code

The _Enso interpreter_ maintains a mapping between its Java code and appropriate
_Enso Code_ internally. It shall be no problem to use that information in the
debugger. All the mentioned IDEs (alas except IntelliJ) support such _mixed
debugging_ of Java and Enso code.

Put a breakpoint into `.enso` file and after a while the breakpoint is hit and
one can inspect variables, step over the _Enso statements_ and more...

![Breakpoint in Enso](https://github.com/enso-org/enso/assets/26887752/54ae4126-f77a-4463-9647-4dd3a5f83526)

...as one can seamlessly switch to debugging on the Enso interpreter itself! One
can place breakpoint into Java class like `PanicException.java` and continue
debugging with `F5`:

![Breakpoint in Java](https://github.com/enso-org/enso/assets/26887752/db3fbe4e-3bb3-4d4a-bb2a-b5039f716c85)

Should one ever want to jump back from Java to Enso one can use the _"Pause in
GraalVM Script"_ action. Select it and continue with `F5` - as soon as the code
reaches a statement in Enso, it stops:

![Pause in GraalVM](https://github.com/enso-org/enso/assets/26887752/98eb0bb7-48c2-4208-9d9a-5b8bacc99de2)

Similar behavior to _VSCode_ is available in _IGV_ and _NetBeans_. Understanding
both the _engine Java code_ as well as _Enso Code_ shall be pretty simple with
these tools. For example _NetBeans_ offers _Toggle Pause in GraalVM Script_
button in the toolbar:

![NetBeans Debugger](https://user-images.githubusercontent.com/26887752/209614191-b0513635-819b-4c64-a6f9-9823b90a1513.png)

and your execution shall stop on the next `.enso` line of code.

These advanced developer tools allow one to debug both - the _Enso code_ as well
as _Java code_. The stack traces shows a _mixture of Java and Enso stack frames_
by default. Right-clicking on the thread allows one to switch to plain Java view
(with a way more stack frames) and back. Analyzing low level details as well as
Enso developer point of view shall be simple with such tool.

### Tips and Tricks (for poor IntelliJ users)

There is no simple mapping of the Enso source code to the engine's Java code, so
if you try to debug a specific expression, it might be a bit tricky. However,
the following steps should help you to skip all the irrelevant code and get to
the code you are interested in:
Finding the mapping of the Enso source code to the engine's Java code in
_IntelliJ_ isn't easy. Trying to find out how to debug a specific expression is
way more tricky than in case of _VSCode_, _IGV_ or _NetBeans_. However, if you
really want to stick with _IntelliJ_ as your only tool, following steps may help
you to skip all the irrelevant code and get to the code you are interested in:

- To get the mapping to Enso source code, evaluate the following expression in
the Java debugger: `this.getRootNode().getSourceSection()`. Note that this,
obviously, works only when the debugger is stopped in some Truffle node code.
- To debug a method called `foo`, put a breakpoint in
`org.enso.interpreter.node.ClosureRootNode#execute` with a condition on
`this.name.contains("foo")`
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,22 @@
import java.io.IOException;
import org.enso.compiler.pass.analyse.AliasAnalysis.Graph;
import org.enso.compiler.pass.resolve.DocumentationComments;
import org.enso.compiler.pass.resolve.DocumentationComments$;
import org.enso.compiler.pass.resolve.ExpressionAnnotations$;
import org.enso.compiler.pass.resolve.FullyQualifiedNames;
import org.enso.compiler.pass.resolve.FullyQualifiedNames$;
import org.enso.compiler.pass.resolve.GenericAnnotations$;
import org.enso.compiler.pass.resolve.GlobalNames$;
import org.enso.compiler.pass.resolve.IgnoredBindings;
import org.enso.compiler.pass.resolve.IgnoredBindings$;
import org.enso.compiler.pass.resolve.MethodCalls$;
import org.enso.compiler.pass.resolve.MethodDefinitions$;
import org.enso.compiler.pass.resolve.ModuleAnnotations;
import org.enso.compiler.pass.resolve.ModuleAnnotations$;
import org.enso.compiler.pass.resolve.Patterns$;
import org.enso.compiler.pass.resolve.TypeNames$;
import org.enso.compiler.pass.resolve.TypeSignatures;
import org.enso.compiler.pass.resolve.TypeSignatures$;
import org.enso.persist.Persistable;
import org.enso.persist.Persistance;
import org.openide.util.lookup.ServiceProvider;
Expand All @@ -30,6 +42,24 @@
@Persistable(clazz = FullyQualifiedNames.FQNResolution.class, id = 1128)
@Persistable(clazz = FullyQualifiedNames.ResolvedLibrary.class, id = 1129)
@Persistable(clazz = FullyQualifiedNames.ResolvedModule.class, id = 1130)
@Persistable(clazz = AliasAnalysis$.class, id = 1201)
@Persistable(clazz = BindingAnalysis$.class, id = 1202)
@Persistable(clazz = CachePreferenceAnalysis$.class, id = 1203)
@Persistable(clazz = DataflowAnalysis$.class, id = 1204)
@Persistable(clazz = GlobalNames$.class, id = 1205)
@Persistable(clazz = IgnoredBindings$.class, id = 1206)
@Persistable(clazz = Patterns$.class, id = 1207)
@Persistable(clazz = TailCall$.class, id = 1208)
@Persistable(clazz = TypeNames$.class, id = 1209)
@Persistable(clazz = TypeSignatures$.class, id = 1210)
@Persistable(clazz = DocumentationComments$.class, id = 1211)
@Persistable(clazz = ModuleAnnotations$.class, id = 1212)
@Persistable(clazz = GatherDiagnostics$.class, id = 1213)
@Persistable(clazz = MethodCalls$.class, id = 1214)
@Persistable(clazz = MethodDefinitions$.class, id = 1215)
@Persistable(clazz = GenericAnnotations$.class, id = 1216)
@Persistable(clazz = ExpressionAnnotations$.class, id = 1217)
@Persistable(clazz = FullyQualifiedNames$.class, id = 1218)
public final class PassPersistance {
private PassPersistance() {}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ import org.enso.compiler.pass.analyse.BindingAnalysis
import org.enso.editions.LibraryName
import org.enso.polyglot.CompilationStage
import scala.collection.mutable
import java.io.IOException

/** Runs imports resolution. Starts from a given module and then recursively
* collects all modules that are reachable from it.
Expand Down Expand Up @@ -47,11 +48,27 @@ class ImportResolver(compiler: Compiler) {

def analyzeModule(current: Module): List[Module] = {
val context = compiler.context
val ir = context.getIr(current)
val currentLocal = ir.unsafeGetMetadata(
BindingAnalysis,
"Non-parsed module used in ImportResolver"
)
val (ir, currentLocal) =
try {
val ir = context.getIr(current)
val currentLocal = ir.unsafeGetMetadata(
BindingAnalysis,
"Non-parsed module used in ImportResolver"
)
(ir, currentLocal)
} catch {
case _: IOException =>
context.updateModule(
current,
u => {
u.ir(null)
u.compilationStage(CompilationStage.INITIAL)
u.invalidateCache()
}
)
compiler.ensureParsed(current)
return analyzeModule(current)
}
// put the list of resolved imports in the module metadata
if (
context
Expand Down Expand Up @@ -156,9 +173,21 @@ class ImportResolver(compiler: Compiler) {
val mod = name.parts.dropRight(1).map(_.name).mkString(".")
compiler.getModule(mod).flatMap { mod =>
compiler.ensureParsed(mod)
mod
.getBindingsMap()
.definedEntities
var b = mod.getBindingsMap()
if (b == null) {
compiler.context.updateModule(
mod,
{ u =>
u.invalidateCache()
u.ir(null)
u.compilationStage(CompilationStage.INITIAL)
}
)
compiler.ensureParsed(mod)
b = mod.getBindingsMap()
}

b.definedEntities
.find(_.name == tp)
.collect { case t: Type =>
ResolvedType(ModuleReference.Concrete(mod), t)
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
package org.enso.compiler.core.ir;

import java.io.IOException;
import java.util.AbstractMap;
import java.util.LinkedHashMap;
import java.util.LinkedHashSet;
import java.util.Map;
import java.util.Objects;
import java.util.Set;

import org.enso.persist.Persistance;
import org.enso.persist.Persistance.Reference;


final class IrLazyMap<K, V> extends AbstractMap<K, V> {
private final Map<K, Entry<K, V>> delegate;

@SuppressWarnings("unchecked")
IrLazyMap(Persistance.Input in) throws IOException {
var map = new LinkedHashMap<K, Entry<K, V>>();
var n = in.readInt();
for (var i = 0; i < n; i++) {
var key = (K) in.readObject();
var ref = (Reference<V>) in.readReference(Object.class);
var en = new En<>(key, ref);
map.put(key, en);
}
this.delegate = map;
}

@Override
public Set<Entry<K, V>> entrySet() {
return new LinkedHashSet<>(this.delegate.values());
}

@Override
public V get(Object key) {
var entry = this.delegate.get(key);
return entry == null ? null : entry.getValue();
}

private static final class En<K, V> implements Entry<K, V> {
private final K key;
private final Reference<V> ref;

En(K key, Reference<V> ref) {
this.key = key;
this.ref = ref;
}

@Override
public K getKey() {
return key;
}

@Override
@SuppressWarnings("unchecked")
public V getValue() {
return (V) ref.get(Object.class);
}

@Override
public V setValue(V value) {
throw new UnsupportedOperationException();
}

@Override
public int hashCode() {
int hash = 7;
hash = 29 * hash + Objects.hashCode(this.key);
return hash;
}

@Override
public boolean equals(Object obj) {
if (this == obj) {
return true;
}
if (obj instanceof En<?, ?> other) {
return Objects.equals(this.key, other.key);
}
return false;
}
}
}
Loading

0 comments on commit 48f9651

Please sign in to comment.