Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade to GraalVM 20.3 #13644

Merged
merged 4 commits into from
Jan 4, 2021
Merged

Upgrade to GraalVM 20.3 #13644

merged 4 commits into from
Jan 4, 2021

Conversation

gsmet
Copy link
Member

@gsmet gsmet commented Dec 2, 2020

Running CI in my fork to see how much fun we will have.

Fixes #13954

@ghost ghost added area/core area/dependencies Pull requests that update a dependency file area/devtools Issues/PR related to maven, gradle, platform and cli tooling/plugins labels Dec 2, 2020
@gsmet
Copy link
Member Author

gsmet commented Dec 3, 2020

Two failures:

The Tika tests are failing with:

Detailed message:
Error: No instances of java.awt.color.ICC_ColorSpace are allowed in the image heap as this class should be initialized at image runtime. To see how this object got instantiated use --trace-object-instantiation=java.awt.color.ICC_ColorSpace.
Trace: Object was reached by 
	reading field java.awt.image.ColorModel.colorSpace of
		constant java.awt.image.DirectColorModel@5c78ebab reached by 
	reading field java.awt.image.ColorModel.RGBdefault
Error: No instances of java.awt.color.ICC_ColorSpace are allowed in the image heap as this class should be initialized at image runtime. To see how this object got instantiated use --trace-object-instantiation=java.awt.color.ICC_ColorSpace.
Trace: Object was reached by 
	reading field org.apache.pdfbox.pdmodel.graphics.color.PDDeviceRGB.awtColorSpace of
		constant org.apache.pdfbox.pdmodel.graphics.color.PDDeviceRGB@538958bf reached by 
	scanning method org.apache.pdfbox.pdmodel.graphics.color.PDICCBased.fallbackToAlternateColorSpace(PDICCBased.java:290)
Call path from entry point to org.apache.pdfbox.pdmodel.graphics.color.PDICCBased.fallbackToAlternateColorSpace(Exception): 
	at org.apache.pdfbox.pdmodel.graphics.color.PDICCBased.fallbackToAlternateColorSpace(PDICCBased.java:288)
	at org.apache.pdfbox.pdmodel.graphics.color.PDICCBased.loadICCProfile(PDICCBased.java:266)
	at org.apache.pdfbox.pdmodel.graphics.color.PDICCBased.<init>(PDICCBased.java:131)
	at org.apache.pdfbox.pdmodel.graphics.color.PDICCBased.create(PDICCBased.java:160)
	at org.apache.pdfbox.pdmodel.graphics.color.PDColorSpace.create(PDColorSpace.java:192)
	at org.apache.pdfbox.pdmodel.graphics.color.PDColorSpace.create(PDColorSpace.java:70)
	at org.apache.pdfbox.pdmodel.graphics.color.PDColorSpace.create(PDColorSpace.java:53)
	at org.apache.pdfbox.pdmodel.graphics.color.PDDeviceNProcess.getColorSpace(PDDeviceNProcess.java:75)
	at org.apache.pdfbox.pdmodel.graphics.color.PDDeviceNProcess.toString(PDDeviceNProcess.java:103)

And also related to ImageIO:

Error:  io.quarkus.it.main.ImageIOITCase.testImageRead  Time elapsed: 0.13 s  <<< FAILURE!
java.lang.AssertionError: 
1 expectation failed.
Response body doesn't match expectation.
Expected: "1x1"
  Actual: <!doctype html>
<html lang="en">
<head>
    <title>Internal Server Error - Caught exception during read: Error id d5cf123d-89fc-4890-9eb0-fe57fd546f98-1</title>

Strangely enough, we don't have a proper Exception in this case.

@gsmet
Copy link
Member Author

gsmet commented Dec 3, 2020

/cc @zakkak @galderz

@zakkak
Copy link
Contributor

zakkak commented Dec 3, 2020

In Mandrel 20.3.0.x this is fixed by graalvm/mandrel#174

I tried patching Quarkus to work with GraalVM CE 20.3 as well but with no success.
You can see my patch here: zakkak@cf3f394
And the failure here: https://github.com/zakkak/quarkus/runs/1415733753#step:7:702

(Fixing #13567 might indeed solve the issues I see as well.)
Edit: fixing #13567 is irrelevant as it is broken in 20.2 as well

@zakkak
Copy link
Contributor

zakkak commented Dec 4, 2020

FTR: The commit that breaks the test is oracle/graal@39d7ff7 (part of GraalVM CE 20.3) and the commit that fixes it is oracle/graal@5d2997a (part of Mandrel 20.3 and GraalVM 21.0 but not GraalVM 20.3).

The first commit (oracle/graal@39d7ff7) forces us to remove any ICC_ColorSpace instances from the native image (by scheduling classes for reinitialization like I do in zakkak@cf3f394). Unfortunately though, this results in the native image to rely on AWTlibawt at runtime:

2020-12-04 19:00:40,120 ERROR [io.qua.ver.htt.run.QuarkusErrorHandler] (executor-thread-1) HTTP Request to /parse/text failed, error id: f85d6761-59c0-4bd9-afc7-8b46505a3e9d-1: org.jboss.resteasy.spi.UnhandledException: java.lang.UnsatisfiedLinkError: no awt in java.library.path
        at org.jboss.resteasy.core.ExceptionHandler.handleApplicationException(ExceptionHandler.java:106)
        at org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:372)
        at org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:218)
        at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:519)
        at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:261)
        at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:161)
        at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:364)
        at org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:164)
        at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:247)
        at io.quarkus.resteasy.runtime.standalone.RequestDispatcher.service(RequestDispatcher.java:73)
        at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler.dispatch(VertxRequestHandler.java:131)
        at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler.access$000(VertxRequestHandler.java:40)
        at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler$1.run(VertxRequestHandler.java:92)
        at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
        at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:2046)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1578)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1452)
        at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)
        at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)
        at java.lang.Thread.run(Thread.java:834)
        at org.jboss.threads.JBossThread.run(JBossThread.java:479)
        at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:519)
        at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)
Caused by: java.lang.UnsatisfiedLinkError: no awt in java.library.path
        at com.oracle.svm.core.jdk.NativeLibrarySupport.loadLibrary(NativeLibrarySupport.java:131)
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:270)
        at java.lang.Runtime.loadLibrary0(Runtime.java:830)
        at java.lang.Runtime.loadLibrary(Runtime.java:232)
        at java.lang.System.loadLibrary(System.java:351)
        at sun.java2d.cmm.lcms.LCMS$1.run(LCMS.java:208)
        at java.security.AccessController.doPrivileged(AccessController.java:84)
        at sun.java2d.cmm.lcms.LCMS.getModule(LCMS.java:202)
        at sun.java2d.cmm.lcms.LcmsServiceProvider.getModule(LcmsServiceProvider.java:34)
        at sun.java2d.cmm.CMMServiceProvider.getColorManagementModule(CMMServiceProvider.java:31)
        at sun.java2d.cmm.CMSManager.getModule(CMSManager.java:68)
        at java.awt.color.ICC_Profile.getInstance(ICC_Profile.java:790)
        at java.awt.color.ICC_Profile.getInstance(ICC_Profile.java:1026)
        at java.awt.color.ICC_Profile.getInstance(ICC_Profile.java:991)
        at java.awt.color.ICC_Profile$2.run(ICC_Profile.java:940)
        at java.awt.color.ICC_Profile$2.run(ICC_Profile.java:936)
        at java.security.AccessController.doPrivileged(AccessController.java:84)
        at java.awt.color.ICC_Profile.getStandardProfile(ICC_Profile.java:935)
        at java.awt.color.ICC_Profile.getDeferredInstance(ICC_Profile.java:1067)
        at java.awt.color.ICC_Profile.getInstance(ICC_Profile.java:854)
        at java.awt.color.ColorSpace.getInstance(ColorSpace.java:298)
        at org.apache.pdfbox.pdmodel.graphics.color.PDDeviceRGB.init(PDDeviceRGB.java:63)
        at org.apache.pdfbox.pdmodel.graphics.color.PDDeviceRGB.toRGBImage(PDDeviceRGB.java:108)
        at org.apache.pdfbox.pdmodel.PDDocument.<clinit>(PDDocument.java:108)
        at com.oracle.svm.core.classinitialization.ClassInitializationInfo.invokeClassInitializer(ClassInitializationInfo.java:351)
        at com.oracle.svm.core.classinitialization.ClassInitializationInfo.initialize(ClassInitializationInfo.java:271)
        at org.apache.tika.parser.pdf.PDFParser.getPDDocument(PDFParser.java:190)
        at org.apache.tika.parser.pdf.PDFParser.parse(PDFParser.java:149)
        at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:280)
        at org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:143)
        at io.quarkus.tika.TikaParser.parseStream(TikaParser.java:85)
        at io.quarkus.tika.TikaParser.parse(TikaParser.java:44)
        at io.quarkus.tika.TikaParser.parse(TikaParser.java:40)
        at io.quarkus.tika.TikaParser.parse(TikaParser.java:32)
        at io.quarkus.it.tika.TikaParserResource.extractText(TikaParserResource.java:29)
        at java.lang.reflect.Method.invoke(Method.java:566)
        at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:170)
        at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:130)
        at org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:643)
        at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:507)
        at org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$2(ResourceMethodInvoker.java:457)
        at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:364)
        at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:459)
        at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:419)
        at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:393)
        at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:68)
        at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:492)
        ... 19 more

@gsmet
Copy link
Member Author

gsmet commented Dec 4, 2020

@zakkak I don't understand your statement here:

Unfortunately though, this results in the native image to rely on AWT at runtime

AFAICS, we already depended on AWT at runtime but, now, somehow the module is not available anymore?

At least that's my understanding of the error?

@zakkak
Copy link
Contributor

zakkak commented Dec 4, 2020

@zakkak I don't understand your statement here:

Unfortunately though, this results in the native image to rely on AWT at runtime

Sorry for that. What actually happens now (GraalVM 20.3 and Quarkus zakkak@cf3f394) that was not happening before (GraalVM 20.2 and Quarkus master) is the invocation of System.loadLibrary("awt") at runtime from sun.java2d.cmm.lcms.LCMS$1.run(LCMS.java:208).

@gsmet
Copy link
Member Author

gsmet commented Dec 4, 2020

It did happen at build time before, then, right?

If so, maybe we should just call it at static init?

(throwing random ideas...)

@zakkak
Copy link
Contributor

zakkak commented Dec 4, 2020

It did happen at build time before, then, right?

Yes and then because the corresponding class was initialized it didn't need to follow that path and load it during runtime.

If so, maybe we should just call it at static init?

(throwing random ideas...)

Loading libawt doesn't help, calling something else that would prevent it from being loaded at runtime should work though, e.g., CMSManager.getModule().

However I am not able to verify it using:

import java.awt.color.ColorSpace;
import sun.java2d.cmm.CMSManager;

public class ColorSpaceTest2 {
    private final static CMSManager.PCMM pcmm;

    static {
        pcmm = CMSManager.getModule();
    }

    public static void main(String[] args) {
        ColorSpace.getInstance(ColorSpace.CS_sRGB);
    }
}

and

/opt/jvms/graalvm-ce-java11-20.3.0/bin/native-image --initialize-at-build-time= ColorSpaceTest2; ./colorspacetest2

It doesn't run the static initializer at build time, any hints on how to force it?

@zakkak
Copy link
Contributor

zakkak commented Dec 5, 2020

@gsmet please have a look at #13718 for the Tika issue.

@gsmet
Copy link
Member Author

gsmet commented Dec 5, 2020

@zakkak cool, thanks! Once you have made the adjustments in the other PR, I will incorporate your commit here and see how it goes.

Any idea about the other failure? The one failing on finding the size of an image for which we don't have a proper exception?

@gsmet
Copy link
Member Author

gsmet commented Dec 5, 2020

I wonder if we have an issue with the error page in native. It's not the first time I don't see any useful exception there.

@gsmet
Copy link
Member Author

gsmet commented Dec 7, 2020

@zakkak so the Integration Tests / Main fail with the exact same issue:

g.UnsatisfiedLinkError: no awt in java.library.path
	at com.oracle.svm.core.jdk.NativeLibrarySupport.loadLibrary(NativeLibrarySupport.java:131)
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:270)
	at java.lang.Runtime.loadLibrary0(Runtime.java:830)
	at java.lang.Runtime.loadLibrary(Runtime.java:232)
	at java.lang.System.loadLibrary(System.java:351)
	at sun.java2d.cmm.lcms.LCMS$1.run(LCMS.java:208)
	at java.security.AccessController.doPrivileged(AccessController.java:84)
	at sun.java2d.cmm.lcms.LCMS.getModule(LCMS.java:202)
	at sun.java2d.cmm.lcms.LcmsServiceProvider.getModule(LcmsServiceProvider.java:34)
	at sun.java2d.cmm.CMMServiceProvider.getColorManagementModule(CMMServiceProvider.java:31)
	at sun.java2d.cmm.CMSManager.getModule(CMSManager.java:68)
	at java.awt.color.ICC_Profile.getInstance(ICC_Profile.java:790)
	at java.awt.color.ICC_Profile.getInstance(ICC_Profile.java:1026)
	at java.awt.color.ICC_Profile.getInstance(ICC_Profile.java:991)
	at java.awt.color.ICC_Profile$2.run(ICC_Profile.java:940)
	at java.awt.color.ICC_Profile$2.run(ICC_Profile.java:936)
	at java.security.AccessController.doPrivileged(AccessController.java:84)
	at java.awt.color.ICC_Profile.getStandardProfile(ICC_Profile.java:935)
	at java.awt.color.ICC_Profile.getDeferredInstance(ICC_Profile.java:1067)
	at java.awt.color.ICC_Profile.getInstance(ICC_Profile.java:854)
	at java.awt.color.ColorSpace.getInstance(ColorSpace.java:298)
	at javax.imageio.ImageTypeSpecifier.&lt;clinit&gt;(ImageTypeSpecifier.java:73)
	at com.oracle.svm.core.classinitialization.ClassInitializationInfo.invokeClassInitializer(ClassInitializationInfo.java:351)
	at com.oracle.svm.core.classinitialization.ClassInitializationInfo.initialize(ClassInitializationInfo.java:271)
	at com.sun.imageio.plugins.png.PNGImageReader.getImageTypes(PNGImageReader.java:1703)
	at com.sun.imageio.plugins.png.PNGImageReader.readImage(PNGImageReader.java:1461)
	at com.sun.imageio.plugins.png.PNGImageReader.read(PNGImageReader.java:1810)
	at javax.imageio.ImageIO.read(ImageIO.java:1468)
	at javax.imageio.ImageIO.read(ImageIO.java:1420)
	at io.quarkus.it.corestuff.ImageIOSupport.doGet(ImageIOSupport.java:18)

What I don't understand is that it was working before so they broke AWT entirely in 20.3?

@zakkak
Copy link
Contributor

zakkak commented Dec 8, 2020

@zakkak so the Integration Tests / Main fail with the exact same issue:

Yes that was expected. Sorry if I didn't make it clear. #13718 fixes only the Apache Tika issue. I will work on the ImageIOTest case next.

What I don't understand is that it was working before so they broke AWT entirely in 20.3?

The loading of libawt at runtime was not working in 20.2 as well. What changed is the set of classes that are allowed to get instantiated in the native-image heap. 20.3 is more strict than 20.2 causing the native image to try and instantiate at runtime some classes that try to load libawt (which is not supported). So the workaround is to try and either avoid instantiating such classes (both at compile and runtime time) or try to force instantiate the class causing the issue at compile time.

Those interested in more details can also check our chat with @gradinac in slack

@zakkak
Copy link
Contributor

zakkak commented Dec 8, 2020

I wonder if we have an issue with the error page in native. It's not the first time I don't see any useful exception there.

How did you finally manage to see the exception you posted in #13644 (comment)?

@gsmet
Copy link
Member Author

gsmet commented Dec 8, 2020

The ugly way :).

The native tests are running in prod mode so you don't see the exceptions.

I went to QuarkusErrorServlet and forced showStack to true.

zakkak and others added 2 commits December 21, 2020 15:37
Also add a comment about removing the PDFBox one when migrating to
GraalVM 21.
@gsmet gsmet marked this pull request as ready for review December 21, 2020 14:48
@gsmet
Copy link
Member Author

gsmet commented Dec 21, 2020

I disabled the ImageIO test.

We will bring them back when upgrading to GraalVM 21 but for now they
don't work with GraalVM 20.3.
Copy link
Contributor

@zakkak zakkak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest removing the changes for Tika (and those from #12912) and disabling both Tika and ImageIO till 20.3.1 gets out (Scheduled for Jan 19) .

20.3.1 will feature the necessary changes to make both tests build and pass without the need of any work-arounds (see oracle/graal#3077)

If 1.11.0.Final can be delayed a bit (currently targeted for Jan 16) we could make sure we have some time to test everything out with 20.3.1 before release

@gsmet
Copy link
Member Author

gsmet commented Dec 22, 2020

The core artifact will be built on January 13th and I don't think we will postpone the release.

But we can upgrade to 20.3.1 in a micro of the 1.11 branch.

As for the Tika workaround, let's keep it, I will revert it when upgrading to 20.3.1.

@gsmet gsmet merged commit cb65f52 into quarkusio:master Jan 4, 2021
@ghost ghost added this to the 1.11 - master milestone Jan 4, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/core area/dependencies Pull requests that update a dependency file area/devtools Issues/PR related to maven, gradle, platform and cli tooling/plugins area/tika
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Upgrade to GraalVM CE 20.3
3 participants