GraalVM-19 fails to build running native-images for Clojure projects

[BrunoBonacci]

Graal version:

openjdk version "1.8.0_212"
OpenJDK Runtime Environment (build 1.8.0_212-20190420112649.buildslave.jdk8u-src-tar--b03)
OpenJDK GraalVM CE 19.0.0 (build 25.212-b03-jvmci-19-b01, mixed mode)

Building a simple main function with Clojure-1.8.0 produces a native image which when executed throws the following error:

$ ~/graalvm-ce-19.0.0/Contents/Home/bin/native-image --report-unsupported-elements-at-runtime -jar ./out/simple-clojure-1.8.0-uberjar.jar -H:Name=./out/simple-clojure-1.8.0-graal-19.0.0
Build on Server(pid: 72432, port: 59638)
[./out/simple-clojure-1.8.0-graal-19.0.0:72432]    classlist:   1,382.83 ms
[./out/simple-clojure-1.8.0-graal-19.0.0:72432]        (cap):   1,120.46 ms
[./out/simple-clojure-1.8.0-graal-19.0.0:72432]        setup:   1,559.23 ms
[./out/simple-clojure-1.8.0-graal-19.0.0:72432]   (typeflow):   6,141.63 ms
[./out/simple-clojure-1.8.0-graal-19.0.0:72432]    (objects):   2,959.83 ms
[./out/simple-clojure-1.8.0-graal-19.0.0:72432]   (features):     258.70 ms
[./out/simple-clojure-1.8.0-graal-19.0.0:72432]     analysis:   9,535.83 ms
[./out/simple-clojure-1.8.0-graal-19.0.0:72432]     (clinit):     217.39 ms
[./out/simple-clojure-1.8.0-graal-19.0.0:72432]     universe:     458.92 ms
[./out/simple-clojure-1.8.0-graal-19.0.0:72432]      (parse):     749.60 ms
[./out/simple-clojure-1.8.0-graal-19.0.0:72432]     (inline):   1,945.61 ms
[./out/simple-clojure-1.8.0-graal-19.0.0:72432]    (compile):   9,276.65 ms
[./out/simple-clojure-1.8.0-graal-19.0.0:72432]      compile:  12,531.96 ms
[./out/simple-clojure-1.8.0-graal-19.0.0:72432]        image:     928.66 ms
[./out/simple-clojure-1.8.0-graal-19.0.0:72432]        write:     262.70 ms
[./out/simple-clojure-1.8.0-graal-19.0.0:72432]      [total]:  26,745.34 ms

$ ./out/simple-clojure-1.8.0-graal-19.0.0
Exception in thread "main" java.lang.ExceptionInInitializerError
	at com.oracle.svm.core.hub.ClassInitializationInfo.initialize(ClassInitializationInfo.java:287)
	at java.lang.Class.ensureInitialized(DynamicHub.java:437)
	at clojure.lang.Namespace.<init>(Namespace.java:34)
	at clojure.lang.Namespace.findOrCreate(Namespace.java:176)
	at clojure.lang.Var.internPrivate(Var.java:151)
	at simple.main.<clinit>(Unknown Source)
	at com.oracle.svm.core.hub.ClassInitializationInfo.invokeClassInitializer(ClassInitializationInfo.java:347)
	at com.oracle.svm.core.hub.ClassInitializationInfo.initialize(ClassInitializationInfo.java:267)
	at java.lang.Class.ensureInitialized(DynamicHub.java:437)
Caused by: java.io.FileNotFoundException: Could not locate clojure/core__init.class or clojure/core.clj on classpath.
	at clojure.lang.RT.load(RT.java:456)
	at clojure.lang.RT.load(RT.java:419)
	at clojure.lang.RT.doInit(RT.java:461)
	at clojure.lang.RT.<clinit>(RT.java:331)
	at com.oracle.svm.core.hub.ClassInitializationInfo.invokeClassInitializer(ClassInitializationInfo.java:347)
	at com.oracle.svm.core.hub.ClassInitializationInfo.initialize(ClassInitializationInfo.java:267)
	... 8 more

This works fine with version: OpenJDK GraalVM CE 1.0.0-rc14 (build 25.202-b08-jvmci-0.56, mixed mode)

I've also cross checked all the latest Clojure versions with both GraalVM versions and this is what I get:

----------------------------------------------------------------------
                      Verifying native images
----------------------------------------------------------------------
verifying simple-clojure-1.10.0-graal-1.0.0-rc14   : OK
verifying simple-clojure-1.9.0-graal-1.0.0-rc14    : OK
verifying simple-clojure-1.8.0-graal-1.0.0-rc14    : OK
verifying simple-clojure-1.7.0-graal-1.0.0-rc14    : OK
verifying simple-clojure-1.10.0-graal-1.0.0-rc15   : OK
verifying simple-clojure-1.9.0-graal-1.0.0-rc15    : OK
verifying simple-clojure-1.8.0-graal-1.0.0-rc15    : OK
verifying simple-clojure-1.7.0-graal-1.0.0-rc15    : OK
verifying simple-clojure-1.10.0-graal-1.0.0-rc16   : OK
verifying simple-clojure-1.9.0-graal-1.0.0-rc16    : OK
verifying simple-clojure-1.8.0-graal-1.0.0-rc16    : OK
verifying simple-clojure-1.7.0-graal-1.0.0-rc16    : OK
verifying simple-clojure-1.10.0-graal-19.0.0       : FAIL
verifying simple-clojure-1.9.0-graal-19.0.0        : FAIL
verifying simple-clojure-1.8.0-graal-19.0.0        : FAIL
verifying simple-clojure-1.7.0-graal-19.0.0        : FAIL

To reproduce the jars here all the steps:
https://github.com/BrunoBonacci/graalvm-clojure/tree/master/clojure

In attachment the standalone JAR
simple-clojure-1.8.0-uberjar.tar.gz

[BrunoBonacci]

After additional tests, it seems that the regression was introduced after GraalVM-CE-1.0.0-rc16 as it is working with this version.

[vjovanov]

We have changed the default for class initialization. Now, application classes are initialized at build time. Unfortunately, this change had to be done to make a larger set of applications work out of the box.

The easy way out is to pass --initialize-at-build-time and all will behave as before.
A proper solution is to define the list of packages that need to be initialized at build time for closure to work and pass it as CSV to --initialize-at-build-time. For example, such a list could look like:

--initialize-at-build-time=clojure.lang

At run time, clojure.lang.RT can't find a file clojure/core__init.class for some reason. Any idea why this could be the case? Does this static initializer do dynamic class loading? If that is the case, we will have to initialize it at build time.

[BrunoBonacci]

Hi @vjovanov ,
thanks for your quick reply. Yes --initialize-at-build-time does seem to fix the problem.
Where can I find more information on what the --initialize-at-build-time does?
How should I select which packages need to initialize a build time vs the run time?

There are a number of other packages in the clojure-1.8.0.jar here is the list:

clojure
clojure/asm
clojure/asm/commons
clojure/core
clojure/core/protocols
clojure/core/proxy$clojure/lang
clojure/data
clojure/inspector/proxy$java/lang
clojure/inspector/proxy$javax/swing/table
clojure/instant/proxy$java/lang
clojure/java
clojure/java/api
clojure/java/browse_ui/proxy$java/lang
clojure/java/io
clojure/lang
clojure/pprint
clojure/pprint/proxy$java/io
clojure/reflect
clojure/reflect/proxy$clojure/asm
clojure/repl/proxy$java/io
clojure/repl/proxy$java/lang
clojure/test
clojure/xml/proxy$java/lang

Do all of these need to be listed in the --initialize-at-build-time flag?

[vjovanov]

We have this document describing how it works. If it is still unclear from the doc and from flags I will expand the document.

[BrunoBonacci]

Great doc thx.

[lvh]

I'm not sure if I should produce a new ticket for this (happy to! I'm also working on a tight reproducer), but one consequence of this is that just setting --initialize-at-build-time makes all HTTPS support break, see #712 (closed but should be reopened, I think) and #1074.

[BrunoBonacci]

Hi @vjovanov ,
thanks again for the link. My understanding is that to get the best performances we should aim to have most of the initializations done a build time. However, some must be forced at runtime.
What is the best way to find the minimal set of classes which initialization needs to be forced a runtime? How to deal with classes which use JNI/JNA libraries?

Thanks

[wololock]

Hi @BrunoBonacci! You could try using native-image-agent GraalVM Java agent to generate the configuration for reflections, proxies or JNI.

https://github.com/oracle/graal/blob/master/substratevm/CONFIGURE.md

All you have to do is to run your Clojure program using regular java using GraalVM's OpenJDK with native-image-agent enabled. I use this approach with my Groovy examples:

https://github.com/wololock/graalvm-groovy-examples/blob/master/hello-world/compile.sh

Maybe this will help you in your investigation.

[BrunoBonacci]

Great, thanks for the doc link and the example, I'll give it a try and feedback on the result.

[wirthi]

I assume this problem has been long resolved, closing this issue. Feel free to reopen if you see a problem with a recent version of GraalVM.