Espresso support with most recent Enso@GraalVM

[JaroslavTulach]

Pull Request Description

Once upon a time there was #4877 which managed to convince Enso native-image to load standard libraries via Espresso. Since then, however, the codebase changed and the integration with GraalVM 22.3.x and Espresso has to look differently right now. This PR updates the original code to most recent develop branch.

Important Notes

Checklist

Please ensure that the following checklist has been satisfied before submitting the PR:

• The documentation has been updated, if necessary.
• All code follows the
  Scala,
  Java,
  and
  Rust
  style guides. In case you are using a language not listed above, follow the Rust style guide.
• All code has been tested:
  • Unit tests have been written where possible.

[JaroslavTulach]

Outdated. Rather read the documentation. Keeping this comment only for historical purposes...

To build and test this PR via sbt (btw. try ./run backend sbt if you want to get all the dependencies):

$ /graalvm-ce-java17-22.3.1/bin/gu install espresso
enso$ sbt --java-home /graalvm-ce-java17-22.3.1/ bootstrap
enso$ sbt --java-home /graalvm-ce-java17-22.3.1/ buildEngineDistribution
enso$ sbt --java-home /graalvm-ce-java17-22.3.1/ engine-runner/buildNativeImage

Let's also create a file h.enso with following content:

import Standard.Base.IO

main = IO.println <| "Hello World!"

Now we can execute the created bits using different styles...

Native Image with Espresso

enso$ JAVA_HOME=/graalvm-ce-java17-22.3.1/ ./runner --run h.enso
Hello World!

Great, it works. Enso can successfully call into Espresso and invoke System.out.println.

JVM with Espresso

enso$ JAVA_HOME=/graalvm-ce-java17-22.3.1/ 
    ./built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/bin/enso --run h.enso

Useful for debugging and finding out what is going on. Pass in additional JAVA_OPTS=-agentlib:jdwp... to debug.

JVM without Espresso

enso$ ENSO_JAVA=hosted JAVA_HOME=/graalvm-ce-java17-22.3.1/ 
    ./built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/bin/enso --run h.enso

Useful to compare the benefits of Espresso over the current (e.g. hosted Java) state.

[JaroslavTulach]

The performance results are interesting. It takes about 1-2s to start Espresso:

And there is a huge difference between following programs:

main = [ "IO", "Hello World!" ]

this one starts almost instantly (as it doesn't need standard libraries, neither Espresso). The version with:

import Standard.Base.IO
main = IO.println <| "Hello World!"

is slow - it needs to prepare the Standard.Base library and also boot Espresso. However also the following version isn't fast:

import Standard.Base.IO
main = [ IO, "Hello World!" ]

and this time it is not problem of Espresso, but of preparing the Standard.Base library. The work on speeding this up such library initialization is tracked as #6100

[JaroslavTulach]

@hubertp this code produces following errors on the console:

enso$ ENSO_JAVA=espresso ./built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/bin/enso --run ~/h.enso 
[ERROR] [2023-09-16T07:24:36+02:00] [enso.org.enso.interpreter.runtime.EnsoContext] Environment variable ENSO_JAVA=espresso, but No language for id java found. Supported languages are: [python, llvm, js, enso]
[ERROR] [2023-09-16T07:24:36+02:00] [enso.org.enso.interpreter.runtime.EnsoContext] Use /graalvm-community-openjdk-17.0.7+7.1/bin/gu install espresso
[ERROR] [2023-09-16T07:24:36+02:00] [enso.org.enso.interpreter.runtime.EnsoContext] Continuing in regular Java mode

are the messages in the right format? I don't find it particularly human readable...

[JaroslavTulach]

With f3705ea one can even run the IDE with ENSO_JAVA=espresso and visualizations seem to work:

Not everything is perfect, Table.new results in something strange, but overall the state isn't completely hopeless.

[JaroslavTulach]

Running test/Test with Espresso

With 513215f and with one more patch (to workaround oracle/graal#7435):

enso$ git diff
diff --git engine/runtime/src/main/java/org/enso/interpreter/node/callable/resolver/HostMethodCallNode.java engine/runtime/src/main/java/org/enso/interpreter/node/callable/resolver/HostMethodCallNode.java
index 32de29eb30..5944927e62 100644
--- engine/runtime/src/main/java/org/enso/interpreter/node/callable/resolver/HostMethodCallNode.java
+++ engine/runtime/src/main/java/org/enso/interpreter/node/callable/resolver/HostMethodCallNode.java
@@ -268,7 +268,7 @@ public abstract class HostMethodCallNode extends Node {
       @Shared("hostValueToEnsoNode") @Cached HostValueToEnsoNode hostValueToEnsoNode) {
     try {
       return hostValueToEnsoNode.execute(instances.instantiate(self, args));
-    } catch (UnsupportedMessageException e) {
+    } catch (Error | UnsupportedMessageException e) {
       CompilerDirectives.transferToInterpreter();
       var ctx = EnsoContext.get(this);
       var err = ctx.getBuiltins().error().makeNotInvokable(self);

I am able to execute whole test/Tests suite:

enso$ .../enso-0.0.0-dev/bin/enso --run test/Tests
1663 tests succeeded.
541 tests failed.

Still a bunch of Espresso bugs to fix, but the result isn't completely desperate.

[enso-bot]

Jaroslav Tulach reports a new STANDUP for yesterday (2023-09-16):

Progress: - ENSO_JAVA=espresso: #6966

• bug for GraalVM team: Espresso doesn't support Value.asValue(IOException).newInstance(...) oracle/graal#7435
• got runEngineDistribution --run test/Tests running with Espresso It should be finished by 2023-09-19.

Next Day: Merge Espresso support

[jdunkerley]

I assume there is no downside on these changes when no in espresso mode.

[4e6]

Just a request to update the SBT build. Otherwise LGTM

[hubertp]

Sometimes we check for JAVA_HOME and sometimes for java.home. Sometimes for both.

[JaroslavTulach]

We should be using System.getProperty("java.home") as that it guaranteed to be set on OpenJDK. JAVA_HOME is just a convenience environment variable used by some (usually shell) scripts to locate the java command.

[enso-bot]

Jaroslav Tulach reports a new STANDUP for yesterday (2023-09-18):

Progress: - running Enso on the latest GraalVM: 2e393dd

• reported: Magnitudes slower up_to.each loop on latest GraalVM #7821
• refactoring IdExecutionService interface & creating builtin to create instrumentation from Enso: Instrumenter to observe behavior of nodes with UUID #7833
• standups, demo meetings, meeting It should be finished by 2023-09-19.

Next Day: Polish & merge Espresso support; work on instrumentation

[enso-bot]

Jaroslav Tulach reports a new STANDUP for yesterday (2023-09-19):

Progress: - ENSO_JAVA=espresso merged: #6966

• testing upgrade of Frgaal: https://github.com/JaroslavTulach/enso/tree/UpdateFrgaal
• investigating GraalVM issues: Magnitudes slower up_to.each loop on latest GraalVM #7821 (comment)
• Avoid StackOverflow merged: Avoid StackOverflow when comparing unknown foreign objects #7780
• emails, reviews, planning meeting, standup It should be finished by 2023-09-19.

Next Day: Discussions; Meta.type; work on instrumentation

GitHub

GitHub - JaroslavTulach/enso at UpdateFrgaal

Hybrid visual and textual functional programming. Contribute to JaroslavTulach/enso development by creating an account on GitHub.