ClassNotFoundException: com.aayushatharva.brotli4j.decoder.DecoderJNI$Wrapper during native build

[ppalaga]

Originally reported in Camel Quarkus apache/camel-quarkus#2692

I wonder if the issue can be reproduced on plain quarkus-google-cloud-services.

It might be related to quarkusio/quarkus@a8a239a

Netty now depends on Brotli4J. Brotli is a compression algorithm.
The implementation uses a native dependency. At the moment, we won't support Brotli in native mode. This
commit adds a substitution to disable it. All decoders using Brotli related classes have been substituted to avoid having to depend on the Brotli4J dependency.

I assume some of the Google cloud libs somehow depends on Brotli4J.

Here are the steps to reproduce from Camel Quarkus

git clone [email protected]:apache/camel-quarkus.git
cd camel-quarkus

# enable the native mode in google cloud tests
git revert ff1bf3a191db256a7a3835715a615dd8e5467d5c

# build the whole repo
mvnd clean install -Dquickly

cd integration-tests/google-storage

# make sure your GRAALVM_HOME points at a GraalVM 21.1 installation
mvn clean verify -Pnative
...
[INFO] --- quarkus-maven-plugin:2.0.0.CR2:build (quarkus-build) @ camel-quarkus-integration-test-google-storage ---
[INFO] [org.jboss.threads] JBoss Threads version 3.4.0.Final
[INFO] [io.quarkus.deployment.pkg.steps.JarResultBuildStep] Building native image source jar: /home/ppalaga/orgs/quarkus/camel-quarkus/integration-tests/google-storage/target/camel-quarkus-integration-test-google-storage-2.0.0-SNAPSHOT-native-image-source-jar/camel-quarkus-integration-test-google-storage-2.0.0-SNAPSHOT-runner.jar
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] Building native image from /home/ppalaga/orgs/quarkus/camel-quarkus/integration-tests/google-storage/target/camel-quarkus-integration-test-google-storage-2.0.0-SNAPSHOT-native-image-source-jar/camel-quarkus-integration-test-google-storage-2.0.0-SNAPSHOT-runner.jar
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] Running Quarkus native-image plugin on GraalVM 21.1.0 Java 11 CE (Java Version 11.0.11+8-jvmci-21.1-b05)
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildRunner] /home/ppalaga/.sdkman/candidates/java/21.1.0.r11-grl/bin/native-image -J-Dsun.nio.ch.maxUpdateArraySize=100 -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-Dvertx.logger-delegate-factory-class-name=io.quarkus.vertx.core.runtime.VertxLogDelegateFactory -J-Dvertx.disableDnsResolver=true -J-Dio.netty.leakDetection.level=DISABLED -J-Dio.netty.allocator.maxOrder=3 -J-Duser.language=en -J-Duser.country=IE -J-Dfile.encoding=UTF-8 --initialize-at-build-time= -H:InitialCollectionPolicy=com.oracle.svm.core.genscavenge.CollectionPolicy\$BySpaceAndTime -H:+JNI -H:+AllowFoldMethods -jar camel-quarkus-integration-test-google-storage-2.0.0-SNAPSHOT-runner.jar -H:FallbackThreshold=0 -H:+ReportExceptionStackTraces -H:-AddAllCharsets -H:EnableURLProtocols=http,https -H:NativeLinkerOption=-no-pie -H:-UseServiceLoaderFeature -H:+StackTrace camel-quarkus-integration-test-google-storage-2.0.0-SNAPSHOT-runner
[camel-quarkus-integration-test-google-storage-2.0.0-SNAPSHOT-runner:56824]    classlist:   2,707.59 ms,  1.68 GB
[camel-quarkus-integration-test-google-storage-2.0.0-SNAPSHOT-runner:56824]        setup:     699.80 ms,  1.68 GB
Fatal error:java.lang.NoClassDefFoundError
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
        at java.base/java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:603)
        at java.base/java.util.concurrent.ForkJoinTask.get(ForkJoinTask.java:1006)
        at com.oracle.svm.hosted.NativeImageGenerator.run(NativeImageGenerator.java:499)
        at com.oracle.svm.hosted.NativeImageGeneratorRunner.buildImage(NativeImageGeneratorRunner.java:370)
        at com.oracle.svm.hosted.NativeImageGeneratorRunner.build(NativeImageGeneratorRunner.java:531)
        at com.oracle.svm.hosted.NativeImageGeneratorRunner.main(NativeImageGeneratorRunner.java:119)
        at com.oracle.svm.hosted.NativeImageGeneratorRunner$JDK9Plus.main(NativeImageGeneratorRunner.java:568)
Caused by: java.lang.NoClassDefFoundError: Lcom/aayushatharva/brotli4j/decoder/DecoderJNI$Wrapper;
        at java.base/java.lang.Class.getDeclaredFields0(Native Method)
        at java.base/java.lang.Class.privateGetDeclaredFields(Class.java:3061)
        at java.base/java.lang.Class.getDeclaredFields(Class.java:2248)
        at com.oracle.svm.hosted.substitute.AnnotationSubstitutionProcessor.handleDeletedClass(AnnotationSubstitutionProcessor.java:531)
        at com.oracle.svm.hosted.substitute.AnnotationSubstitutionProcessor.handleClass(AnnotationSubstitutionProcessor.java:306)
        at com.oracle.svm.hosted.substitute.AnnotationSubstitutionProcessor.init(AnnotationSubstitutionProcessor.java:266)
        at com.oracle.svm.hosted.NativeImageGenerator.createDeclarativeSubstitutionProcessor(NativeImageGenerator.java:954)
        at com.oracle.svm.hosted.NativeImageGenerator.setupNativeImage(NativeImageGenerator.java:888)
        at com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:580)
        at com.oracle.svm.hosted.NativeImageGenerator.lambda$run$2(NativeImageGenerator.java:495)
        at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1407)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
        at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
Caused by: java.lang.ClassNotFoundException: com.aayushatharva.brotli4j.decoder.DecoderJNI$Wrapper
        at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
        ... 16 more

[ppalaga]

See also https://quarkusio.zulipchat.com/#narrow/stream/187038-dev/topic/brotli4j

[ppalaga]

The root cause seems to be that --report-unsupported-elements-at-runtime (available in META-INF/native-image/com.google.cloud/google-cloud-graalvm-support/native-image.properties ) is incompatible with @Delete brought by quarkus-netty.

We need either
A. to get GoogleCloudPlatform/native-image-support-java#141 fixed or
B. we need to stop depending on google-cloud-graalvm-support

@loicmathieu would you have any problem with B. It looks like we could re-create their config using Quarkus means in quarkus-google-cloud-services. WDYT?

[ppalaga]

@gsmet came up with another solution https://github.com/quarkusio/quarkus/pull/17604/files - it works well for Camel Quarkus

[gsmet]

While it works around the issue, I would say it's still a good idea to start the work to avoir using --report-unsupported-elements-at-runtime.

We might not be as lucky next time. Also it's definitely not a recommended option.

[loicmathieu]

I was going also to said this is an issue on Quarkus netty and not specific to this extension but it seems to be a little more complicated.

Anyhow, I added a comment on Peter's issue on Google Cloud native support, I'm pretty sure we can work with them on this subject ;)

[loicmathieu]

@ppalaga

@loicmathieu would you have any problem with B. It looks like we could re-create their config using Quarkus means in quarkus-google-cloud-services. WDYT?

Yes, using the native support from upstream is the recommended way inside Quarkus when available and it's also the way we work together with the Google Cloud team. Using the library and improving it will also improve native support for the all open source community (not only Quarkus) so it's worth it ;)

[dzou]

I will try to remove the --report-unsupported-elements-at-runtime on our end. We have some blockers as discussed in GoogleCloudPlatform/native-image-support-java#141 (comment)
Primarily due to incorrect shading of a native-image config from a dependency in an upstream library. I will try to implement a fix, but because of the upstream dependency it may take some time.

The root cause seems to be that --report-unsupported-elements-at-runtime (available in META-INF/native-image/com.google.cloud/google-cloud-graalvm-support/native-image.properties ) is incompatible with @Delete brought by quarkus-netty.

We need either
A. to get GoogleCloudPlatform/google-cloud-graalvm-support#141 fixed or
B. we need to stop depending on google-cloud-graalvm-support

I'm curious why these are the only choices. Can GraalVM not be improved so that @Delete works with --report-unsupported-elements-at-runtime?

On a separate note, I understand --report-unsupported-elements-at-runtime is recommended to be dev-only. However, in practice many libraries ultimately use this flag because the requirements imposed by GraalVM are too strict without it. This is one reason why Spring includes the flag by default in their native-image support:
https://docs.spring.io/spring-native/docs/current/reference/htmlsingle/#native-image-options-default

Currently GraalVM is relatively new and libraries are just starting to implement adoption for it. I think in this early stage, you are bound to hit problems that may require --report-unsupported-elements-at-runtime, so I don't think it's clear-cut that this flag "must be removed". It would be more reasonable for GraalVM features to be made compatible with its inclusion.

[loicmathieu]

@gsmet, we discussed this recently, this is a valid point, GraalVM support could be hard and as more and more library will include support for it we may hit this issue more often.

Maybe an issue should be openned on GrallVM side to support @Remove with --report-unsupported-elements-at-runtime or to have a way to override the flag on the command line (to override a value from the config file).

[ppalaga]

I had the same question and asked it here: oracle/graal#1725 (comment)
You may want to speak up there too.

[dzou]

I had the same question and asked it here: oracle/graal#1725 (comment)
You may want to speak up there too.

Gotcha, will do.

In any case, we'll still work on getting the --report-unsupported-elements-at-runtime removed to avoid problems in the future.

[loicmathieu]

This has been worked around in 2.0.0.CR3 but I keep this one openned to track the upstream changes.

[dzou]

This has been worked around in 2.0.0.CR3 but I keep this one openned to track the upstream changes.

Sounds good. We are starting work to remove --report-unsupported-elements-at-runtime. Will keep you updated.

[loicmathieu]

Thanks to @dzou and its team, the --report-unsupported-elements-at-runtime option has been removed on release https://github.com/GoogleCloudPlatform/google-cloud-graalvm-support/releases/tag/0.6.0.

So we can close this one.