[Mandrel 23.0.0] google-bigquery integration test native compilation fails #5006

jamesnetherton · 2023-06-22T07:13:14Z

Bug description

The initial CI fails seems to suggest there are some items that need reflective class configuration:

2023-06-21T13:20:25.2633284Z Caused by: java.lang.IllegalArgumentException: field private java.util.List com.google.api.services.bigquery.model.TableDataList.rows
2023-06-21T13:20:25.2633909Z 	at com.google.api.client.json.JsonParser.parseValue(JsonParser.java:900)
2023-06-21T13:20:25.2634647Z 	at com.google.api.client.json.JsonParser.parse(JsonParser.java:451)
2023-06-21T13:20:25.2635157Z 	at com.google.api.client.json.JsonParser.parseValue(JsonParser.java:787)
2023-06-21T13:20:25.2635519Z 	... 56 more
2023-06-21T13:20:25.2636002Z Caused by: java.lang.IllegalArgumentException: field private java.util.List com.google.api.services.bigquery.model.TableDataList.rows
2023-06-21T13:20:25.2636636Z 	at com.google.api.client.json.JsonParser.parseValue(JsonParser.java:900)
2023-06-21T13:20:25.2637148Z 	at com.google.api.client.json.JsonParser.parseArray(JsonParser.java:641)
2023-06-21T13:20:25.2637635Z 	at com.google.api.client.json.JsonParser.parseValue(JsonParser.java:744)
2023-06-21T13:20:25.2637990Z 	... 58 more
2023-06-21T13:20:25.2638495Z Caused by: java.lang.IllegalArgumentException: field private java.util.List com.google.api.services.bigquery.model.TableRow.f
2023-06-21T13:20:25.2639091Z 	at com.google.api.client.json.JsonParser.parseValue(JsonParser.java:900)
2023-06-21T13:20:25.2639697Z 	at com.google.api.client.json.JsonParser.parse(JsonParser.java:451)
2023-06-21T13:20:25.2640185Z 	at com.google.api.client.json.JsonParser.parseValue(JsonParser.java:787)
2023-06-21T13:20:25.2640541Z 	... 60 more
2023-06-21T13:20:25.2640987Z Caused by: java.lang.IllegalArgumentException: field private java.util.List com.google.api.services.bigquery.model.TableRow.f
2023-06-21T13:20:25.2641583Z 	at com.google.api.client.json.JsonParser.parseValue(JsonParser.java:900)
2023-06-21T13:20:25.2642090Z 	at com.google.api.client.json.JsonParser.parseArray(JsonParser.java:641)
2023-06-21T13:20:25.2642597Z 	at com.google.api.client.json.JsonParser.parseValue(JsonParser.java:744)
2023-06-21T13:20:25.2642932Z 	... 62 more
2023-06-21T13:20:25.2643475Z Caused by: java.lang.IllegalArgumentException: unable to create new instance of class com.google.api.services.bigquery.model.TableCell because it has no accessible default constructor
2023-06-21T13:20:25.2644200Z 	at com.google.api.client.util.Types.handleExceptionForNewInstance(Types.java:162)
2023-06-21T13:20:25.2644728Z 	at com.google.api.client.util.Types.newInstance(Types.java:117)
2023-06-21T13:20:25.2645196Z 	at com.google.api.client.util.Data.newMapInstance(Data.java:550)
2023-06-21T13:20:25.2645681Z 	at com.google.api.client.json.JsonParser.parseValue(JsonParser.java:766)
2023-06-21T13:20:25.2646026Z 	... 64 more
2023-06-21T13:20:25.2646415Z Caused by: java.lang.InstantiationException: com.google.api.services.bigquery.model.TableCell
2023-06-21T13:20:25.2646918Z 	at [email protected]/java.lang.Class.newInstance(DynamicHub.java:639)
2023-06-21T13:20:25.2647359Z 	at com.google.api.client.util.Types.newInstance(Types.java:113)
2023-06-21T13:20:25.2647665Z 	... 66 more
2023-06-21T13:20:25.2648091Z Caused by: java.lang.NoSuchMethodException: com.google.api.services.bigquery.model.TableCell.<init>()
2023-06-21T13:20:25.2648610Z 	at [email protected]/java.lang.Class.checkMethod(DynamicHub.java:1038)
2023-06-21T13:20:25.2649031Z 	at [email protected]/java.lang.Class.getConstructor0(DynamicHub.java:1204)
2023-06-21T13:20:25.2649445Z 	at [email protected]/java.lang.Class.newInstance(DynamicHub.java:626)
2023-06-21T13:20:25.2649747Z 	... 67 more

The text was updated successfully, but these errors were encountered:

jamesnetherton · 2023-06-22T07:16:43Z

The strange thing is, that I cannot replicate the above issue. I have so far got two entirely different errors:

First this:

Failed generating 'camel-quarkus-integration-test-google-bigquery-3.0.0-SNAPSHOT-runner' after 2m 11s.

The build process encountered an unexpected error:

java.lang.RuntimeException: There was an error linking the native image: Linker command exited with 1

Linker command executed:
/usr/bin/gcc -z noexecstack -Wl,--gc-sections -Wl,--version-script,/tmp/SVM-13436729235266490940/exported_symbols.list -Wl,-x -o /project/camel-quarkus-integration-test-google-bigquery-3.0.0-SNAPSHOT-runner camel-quarkus-integration-test-google-bigquery-3.0.0-SNAPSHOT-runner.o /opt/mandrel/lib/svm/clibraries/linux-amd64/liblibchelper.a /opt/mandrel/lib/static/linux-amd64/glibc/libnet.a /opt/mandrel/lib/static/linux-amd64/glibc/libnio.a /opt/mandrel/lib/static/linux-amd64/glibc/libjava.a /opt/mandrel/lib/static/linux-amd64/glibc/libfdlibm.a /opt/mandrel/lib/static/linux-amd64/glibc/libzip.a /opt/mandrel/lib/svm/clibraries/linux-amd64/libjvm.a -Wl,--export-dynamic -v -L/tmp/SVM-13436729235266490940 -L/opt/mandrel/lib/static/linux-amd64/glibc -L/opt/mandrel/lib/svm/clibraries/linux-amd64 -lz -lpthread -ldl -lrt

Linker command output:
Using built-in specs.
COLLECT_GCC=/usr/bin/gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-redhat-linux/8/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-redhat-linux
Configured with: ../configure --enable-bootstrap --enable-languages=c,c++,fortran,lto --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-shared --enable-threads=posix --enable-checking=release --enable-multilib --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-gcc-major-version-only --with-linker-hash-style=gnu --enable-plugin --enable-initfini-array --with-isl --disable-libmpx --enable-offload-targets=nvptx-none --without-cuda-driver --enable-gnu-indirect-function --enable-cet --with-tune=generic --with-arch_32=x86-64 --build=x86_64-redhat-linux
Thread model: posix
gcc version 8.5.0 20210514 (Red Hat 8.5.0-18) (GCC) 
COMPILER_PATH=/usr/libexec/gcc/x86_64-redhat-linux/8/:/usr/libexec/gcc/x86_64-redhat-linux/8/:/usr/libexec/gcc/x86_64-redhat-linux/:/usr/lib/gcc/x86_64-redhat-linux/8/:/usr/lib/gcc/x86_64-redhat-linux/
LIBRARY_PATH=/usr/lib/gcc/x86_64-redhat-linux/8/:/usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64/:/lib/../lib64/:/usr/lib/../lib64/:/usr/lib/gcc/x86_64-redhat-linux/8/../../../:/lib/:/usr/lib/
COLLECT_GCC_OPTIONS='-z' 'noexecstack' '-o' '/project/camel-quarkus-integration-test-google-bigquery-3.0.0-SNAPSHOT-runner' '-v' '-L/tmp/SVM-13436729235266490940' '-L/opt/mandrel/lib/static/linux-amd64/glibc' '-L/opt/mandrel/lib/svm/clibraries/linux-amd64' '-mtune=generic' '-march=x86-64'
 /usr/libexec/gcc/x86_64-redhat-linux/8/collect2 -plugin /usr/libexec/gcc/x86_64-redhat-linux/8/liblto_plugin.so -plugin-opt=/usr/libexec/gcc/x86_64-redhat-linux/8/lto-wrapper -plugin-opt=-fresolution=/tmp/ccvtgKLh.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s --build-id --no-add-needed --eh-frame-hdr --hash-style=gnu -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o /project/camel-quarkus-integration-test-google-bigquery-3.0.0-SNAPSHOT-runner -z noexecstack /usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64/crt1.o /usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64/crti.o /usr/lib/gcc/x86_64-redhat-linux/8/crtbegin.o -L/tmp/SVM-13436729235266490940 -L/opt/mandrel/lib/static/linux-amd64/glibc -L/opt/mandrel/lib/svm/clibraries/linux-amd64 -L/usr/lib/gcc/x86_64-redhat-linux/8 -L/usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/usr/lib/gcc/x86_64-redhat-linux/8/../../.. --gc-sections --version-script /tmp/SVM-13436729235266490940/exported_symbols.list -x camel-quarkus-integration-test-google-bigquery-3.0.0-SNAPSHOT-runner.o /opt/mandrel/lib/svm/clibraries/linux-amd64/liblibchelper.a /opt/mandrel/lib/static/linux-amd64/glibc/libnet.a /opt/mandrel/lib/static/linux-amd64/glibc/libnio.a /opt/mandrel/lib/static/linux-amd64/glibc/libjava.a /opt/mandrel/lib/static/linux-amd64/glibc/libfdlibm.a /opt/mandrel/lib/static/linux-amd64/glibc/libzip.a /opt/mandrel/lib/svm/clibraries/linux-amd64/libjvm.a --export-dynamic -lz -lpthread -ldl -lrt -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/lib/gcc/x86_64-redhat-linux/8/crtend.o /usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64/crtn.o
camel-quarkus-integration-test-google-bigquery-3.0.0-SNAPSHOT-runner.o:(.data+0x3f0): undefined reference to `Java_com_sun_management_internal_OperatingSystemImpl_initialize0'
collect2: error: ld returned 1 exit status
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.image.NativeImageViaCC.handleLinkerFailure(NativeImageViaCC.java:204)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.image.NativeImageViaCC.runLinkerCommand(NativeImageViaCC.java:151)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.image.NativeImageViaCC.write(NativeImageViaCC.java:117)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:724)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.run(NativeImageGenerator.java:539)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.buildImage(NativeImageGeneratorRunner.java:408)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.build(NativeImageGeneratorRunner.java:612)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.start(NativeImageGeneratorRunner.java:134)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.main(NativeImageGeneratorRunner.java:94)

Then that problem magically stopped happening and I now have:

Failed generating 'camel-quarkus-integration-test-google-bigquery-3.0.0-SNAPSHOT-runner' after 42.3s.

The build process encountered an unexpected error:

java.lang.NullPointerException: Cannot invoke "java.util.Set.iterator()" because "triggers" is null
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.ReachabilityHandlerFeature.processReachable(ReachabilityHandlerFeature.java:173)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.ReachabilityHandlerFeature.duringAnalysis(ReachabilityHandlerFeature.java:128)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.lambda$runPointsToAnalysis$10(NativeImageGenerator.java:770)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.FeatureHandler.forEachFeature(FeatureHandler.java:86)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.lambda$runPointsToAnalysis$11(NativeImageGenerator.java:770)
        at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.AbstractAnalysisEngine.runAnalysis(AbstractAnalysisEngine.java:179)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.runPointsToAnalysis(NativeImageGenerator.java:767)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:582)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.run(NativeImageGenerator.java:539)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.buildImage(NativeImageGeneratorRunner.java:408)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.build(NativeImageGeneratorRunner.java:612)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.start(NativeImageGeneratorRunner.java:134)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.main(NativeImageGeneratorRunner.java:94)

Which I see is also happening on the quarkus-google-cloud-services nightly CI:

https://github.com/quarkiverse/quarkus-google-cloud-services/actions/runs/5341794760/jobs/9682960330

zhfeng · 2023-06-27T03:10:44Z

I think there might be something wrong during the graal perform static analysis.

The error about undefined reference to 'Java_com_sun_management_internal_OperatingSystemImpl_initialize0' looks related to this commit

The OPERATING_SYSTEM_IMPL class is not reachable but actually it is. Then sometimes we get the native image build successful and sometimes not. I have no any idea about how the static analysis work here. @zakkak any hint?

jamesnetherton · 2023-06-28T11:05:59Z

This issue is really weird. I have run the build multiple times and got different results.

Run 1: undefined reference to Java_com_sun_management_internal_OperatingSystemImpl_initialize0' collect2: error: ld returned 1 exit status

Run 2: java.lang.NullPointerException: Cannot invoke "java.util.Set.iterator()" because "triggers" is null

Run 3: Tests run: 6, Failures: 0, Errors: 0, Skipped: 0 BUILD SUCCESS

zhfeng · 2023-06-28T11:13:47Z

You have got a success build? Add some reflective class configurations?

jamesnetherton · 2023-06-28T11:31:45Z

You have got a success build? Add some reflective class configurations?

I got a successful build without adding anything new.

zakkak · 2023-06-28T11:47:59Z

Yes, I see the same as @jamesnetherton still haven't figure out what triggers it though. It looks like a data race so far...

zakkak · 2023-06-28T14:07:47Z

As discussed in oracle/graal#6904 the issue is caused by com.google.api:gax-grpc passing -H:-RunReachabilityHandlersConcurrently to native-image.

Ultimately this should be fixed both in GraalVM/Mandrel (see oracle/graal#6904) as well as in com.google.api:gax-grpc (see googleapis/sdk-platform-java#1344).

Till then, the fastest work around is to pass -Dquarkus.native.additional-build-args=-H:+RunReachabilityHandlersConcurrently.

jamesnetherton · 2023-06-28T14:15:27Z

Thanks @zakkak!

Till then, the fastest work around is to pass -Dquarkus.native.additional-build-args=-H:+RunReachabilityHandlersConcurrently.

I guess that'll be something we have to add into our Google extension docs.

AFAIK, there is no way for extensions to provide arbitrary native build args (E.g via a BuildItem), right?

zakkak · 2023-06-28T14:19:55Z

That's correct, I ~~will prepare~~ prepared a Quarkus patch for it though, see quarkusio/quarkus#34386.

Till then, the fastest work around is to pass -Dquarkus.native.additional-build-args=-H:+RunReachabilityHandlersConcurrently

The main issue is that passing -Dquarkus.native.additional-build-args=-H:+RunReachabilityHandlersConcurrently also requires adding more dependencies as discussed in googleapis/sdk-platform-java#1344

zhfeng · 2023-06-28T15:07:56Z

Thanks @zakkak !

I just tried with -Dquarkus.native.additional-build-args=-H:+RunReachabilityHandlersConcurrently but it is still not working. Is there anything I missed or what dependecies should be added?

zakkak · 2023-06-28T17:56:25Z

Not sure what dependencies are needed. I assume you are getting an UnresolvedElementException which should include information about which classes are missing.

https://quarkus.io/guides/native-reference#i-get-a-analysiserrorparsingerror-when-building-a-native-executable-due-to-an-unresolvedelementexception-what-can-i-do has some more info.

zhfeng · 2023-06-29T00:55:55Z

Well, I still got some random results with

java.lang.NullPointerException: Cannot invoke "java.util.Set.iterator()" because "triggers" is null
undefined reference to 'Java_com_sun_management_internal_OperatingSystemImpl_initialize0' collect2: error: ld returned 1 exit status
some refelection errors

zakkak · 2023-06-29T07:40:06Z

@zhfeng , this doesn't sound right. Can you please paste the output following the line

[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] Running Quarkus native-image plugin on native-image 17.0.7 2023-04-18

FWIW I tested with:

./mvnw clean install -Pnative -pl integration-tests/google-bigquery -Dquarkus.native.additional-build-args=-H:+RunReachabilityHandlersConcurrently

zhfeng · 2023-06-29T07:47:56Z

Yeah, I added <quarkus.native.additional-build-args>-H:+RunReachabilityHandlersConcurrently</quarkus.native.additional-build-args> in integration-tests/google-bigquery/pom.xml

The output is like

[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] Running Quarkus native-image plugin on native-image 17.0.7 2023-04-18
OpenJDK Runtime Environment Mandrel-23.0.0.0-Final (build 17.0.7+7)
OpenJDK 64-Bit Server VM Mandrel-23.0.0.0-Final (build 17.0.7+7, mixed mode)
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildRunner] docker run --env LANG=C --rm --user 1000:1000 -v /home/zhfeng/work/zhfeng/camel-quarkus/integration-tests/google-bigquery/target/camel-quarkus-integration-test-google-bigquery-3.0.0-SNAPSHOT-native-image-source-jar:/project:z --name build-native-ueGYN quay.io/quarkus/ubi-quarkus-mandrel-builder-image:jdk-17 -J-Dsun.nio.ch.maxUpdateArraySize=100 -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-Dlogging.initial-configurator.min-level=500 -J-Dvertx.logger-delegate-factory-class-name=io.quarkus.vertx.core.runtime.VertxLogDelegateFactory -J-Dvertx.disableDnsResolver=true -J-Dio.netty.leakDetection.level=DISABLED -J-Dio.netty.allocator.maxOrder=3 -J-Duser.language=zh -J-Duser.country=CN -J-Dfile.encoding=UTF-8 --features=io.quarkus.runner.Feature,io.quarkus.runtime.graal.DisableLoggingFeature -J--add-exports=java.security.jgss/sun.security.krb5=ALL-UNNAMED -J--add-opens=java.base/java.text=ALL-UNNAMED -J--add-opens=java.base/java.io=ALL-UNNAMED -J--add-opens=java.base/java.lang.invoke=ALL-UNNAMED -J--add-opens=java.base/java.util=ALL-UNNAMED -H:BuildOutputJSONFile=camel-quarkus-integration-test-google-bigquery-3.0.0-SNAPSHOT-runner-build-output-stats.json -H:+RunReachabilityHandlersConcurrently -H:+AllowFoldMethods -J-Djava.awt.headless=true --no-fallback --link-at-build-time -H:+ReportExceptionStackTraces -H:-AddAllCharsets --enable-url-protocols=http -H:-UseServiceLoaderFeature -H:+StackTrace -J--add-exports=org.graalvm.sdk/org.graalvm.nativeimage.impl=ALL-UNNAMED --exclude-config io\.netty\.netty-codec /META-INF/native-image/io\.netty/netty-codec/generated/handlers/reflect-config\.json --exclude-config io\.netty\.netty-handler /META-INF/native-image/io\.netty/netty-handler/generated/handlers/reflect-config\.json camel-quarkus-integration-test-google-bigquery-3.0.0-SNAPSHOT-runner -jar camel-quarkus-integration-test-google-bigquery-3.0.0-SNAPSHOT-runner.jar
========================================================================================================================
GraalVM Native Image: Generating 'camel-quarkus-integration-test-google-bigquery-3.0.0-SNAPSHOT-runner' (executable)...
========================================================================================================================
[1/8] Initializing...                                                                                    (7.2s @ 0.34GB)
 Java version: 17.0.7+7, vendor version: Mandrel-23.0.0.0-Final
 Graal compiler: optimization level: 2, target machine: x86-64-v3
 C compiler: gcc (redhat, x86_64, 8.5.0)
 Garbage collector: Serial GC (max heap size: 80% of RAM)
 8 user-specific feature(s)
 - com.google.api.gax.grpc.nativeimage.GrpcNettyFeature
 - com.google.api.gax.grpc.nativeimage.ProtobufMessageFeature
 - com.google.api.gax.nativeimage.GoogleJsonClientFeature
 - com.google.api.gax.nativeimage.OpenCensusFeature
 - com.oracle.svm.thirdparty.gson.GsonFeature
 - io.quarkus.runner.Feature: Auto-generated class by Quarkus from the existing extensions
 - io.quarkus.runtime.graal.DisableLoggingFeature: Disables INFO logging during the analysis phase
 - org.eclipse.angus.activation.nativeimage.AngusActivationFeature
Warning: RecomputeFieldValue.FieldOffset automatic substitution failed. The automatic substitution registration was attempted because a call to sun.misc.Unsafe.objectFieldOffset(Field) was detected in the static initializer of com.google.protobuf.UnsafeUtil. Detailed failure reason(s): The argument of sun.misc.Unsafe.objectFieldOffset(java.lang.reflect.Field) is not a constant value or a field load that can be constant-folded., Could not determine the field where the value produced by the call to sun.misc.Unsafe.objectFieldOffset(Field) for the field offset computation is stored. The call is not directly followed by a field store or by a sign extend node followed directly by a field store. 
[2/8] Performing analysis...  []                                                                        (25.0s @ 3.50GB)
  16,967 (84.92%) of 19,981 types reachable
  25,097 (60.48%) of 41,495 fields reachable
  86,267 (62.59%) of 137,820 methods reachable
   5,448 types, 1,267 fields, and 8,656 methods registered for reflection

------------------------------------------------------------------------------------------------------------------------
                        2.5s (7.5% of total time) in 28 GCs | Peak RSS: 5.53GB | CPU load: 12.11
------------------------------------------------------------------------------------------------------------------------
Produced artifacts:
 /project/svm_err_b_20230629T074513.723_pid74.md (build_info)
========================================================================================================================
Failed generating 'camel-quarkus-integration-test-google-bigquery-3.0.0-SNAPSHOT-runner' after 32.5s.

The build process encountered an unexpected error:

java.lang.NullPointerException: Cannot invoke "java.util.Set.iterator()" because "triggers" is null
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.ReachabilityHandlerFeature.processReachable(ReachabilityHandlerFeature.java:173)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.ReachabilityHandlerFeature.duringAnalysis(ReachabilityHandlerFeature.java:128)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.lambda$runPointsToAnalysis$10(NativeImageGenerator.java:770)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.FeatureHandler.forEachFeature(FeatureHandler.java:86)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.lambda$runPointsToAnalysis$11(NativeImageGenerator.java:770)
	at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.AbstractAnalysisEngine.runAnalysis(AbstractAnalysisEngine.java:179)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.runPointsToAnalysis(NativeImageGenerator.java:767)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:582)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.run(NativeImageGenerator.java:539)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.buildImage(NativeImageGeneratorRunner.java:408)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.build(NativeImageGeneratorRunner.java:612)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.start(NativeImageGeneratorRunner.java:134)
	at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.main(NativeImageGeneratorRunner.java:94)

zhfeng · 2023-06-29T07:50:33Z

I just test with your command but it is still the same.

zakkak · 2023-06-29T08:10:30Z

I just test with your command but it is still the same.

Oh, you are right. I was testing with 999-SNAPSHOT not 3.2.0.CR1.

And it turns out that with 999-SNAPSHOT I am getting an Unresolved Element exception irrespective of the value of RunReachabilityHandlersConcurrently. Back to it :/

zakkak · 2023-06-29T08:46:35Z

Unfortunately the only work around I can think of is to introduce a native-image.properties file containing the following line:

Args = -H:+RunReachabilityHandlersConcurrently

but this file needs to be in the application, e.g. integration-tests/google-bigquery/src/main/resources/META-INF/native-image/native-image.properties, putting it under extensions/google-bigquery won't do the trick since native-image may apply it before applying the corresponding file from gax-grpc...

Now for an actual fix it looks like we would need a gax-grpc extension that would:

Introduce an ExcludeConfigBuildItem to exclude the gax-grpc configuration.
Introduce the appropriate BuildItems to replicate the original gax-grpc configuration, excluding the RunReachabilityHandlersConcurrently part. I.e., register the necessary features and the necessary classes for runtime initialization.

zhfeng · 2023-06-29T08:54:04Z

Oh, good catch! I just wonder why gax-grpc needs -H:-RunReachabilityHandlersConcurrently and it could be fixed in graal in the new release?

zhfeng · 2023-06-29T09:08:08Z

I can confirm that the work around works perfectly!

So @jamesnetherton cam we use this work around for all google related extensions?

zhfeng · 2023-06-30T08:52:48Z

OK, I found another work around to exlucde org.apache.arrow:arrow-memory-netty since it is a runtime dependency.

@jamesnetherton I'm not familar with google-bigquery but it seems we have no any test for apache arrow memory?

This reverts commit 14ad98d.

ppalaga · 2023-07-03T15:43:31Z

@zakkak, I see that quarkus.native.additional-build-args is somehow not powerful enough to override the value present in gax-grpc jar. But it is still not clear to me why? Could you perhaps shed some light on that?

zakkak · 2023-07-03T16:34:55Z

@zakkak, I see that quarkus.native.additional-build-args is somehow not powerful enough to override the value present in gax-grpc jar. But it is still not clear to me why? Could you perhaps shed some light on that?

If you run native-image with --verbose you can see the options that end up being passed to the native image builder after combining options given by the user, the libraries, and the application. These options are passed in the following order:

Options automatically passed by Quarkus
User options passed through quarkus.native.additional-build-args (these can override the ones in 1)
Options automatically passed by Quarkus that we don't want to allow the users to override (see https://github.com/quarkusio/quarkus/blob/5382a3de02facc6f1be2f55d8fdb7d5c1fce8dc2/core/deployment/src/main/java/io/quarkus/deployment/pkg/steps/NativeImageBuildStep.java#L795-L799)
Options passed through native-image.properties of dependencies
Options passed through native-image.properties of the application

with each category being able to override the previous categories. So, since quarkus.native.additional-build-args are applied before the native-image.properties there is no way for them to override the values passed through native-image.properties. AFAIK there is no way to make command line options to apply after options defined in native-image.properties.

HTH

ppalaga · 2023-07-03T19:43:33Z

@zakkak, I see that quarkus.native.additional-build-args is somehow not powerful enough to override the value present in gax-grpc jar. But it is still not clear to me why? Could you perhaps shed some light on that?

If you run native-image with --verbose you can see the options that end up being passed to the native image builder after combining options given by the user, the libraries, and the application. These options are passed in the following order:

Options automatically passed by Quarkus

User options passed through quarkus.native.additional-build-args (these can override the ones in 1)

Options automatically passed by Quarkus that we don't want to allow the users to override (see https://github.com/quarkusio/quarkus/blob/5382a3de02facc6f1be2f55d8fdb7d5c1fce8dc2/core/deployment/src/main/java/io/quarkus/deployment/pkg/steps/NativeImageBuildStep.java#L795-L799)

Options passed through native-image.properties of dependencies

Options passed through native-image.properties of the application

with each category being able to override the previous categories. So, since quarkus.native.additional-build-args are applied before the native-image.properties there is no way for them to override the values passed through native-image.properties. AFAIK there is no way to make command line options to apply after options defined in native-image.properties.

Thanks a lot for the informative answer, @zakkak!

This reverts commit 14ad98d.

…fix for PooledByteBufAllocatorL.java

This reverts commit 14ad98d.

…r PooledByteBufAllocatorL.java

See apache/camel-quarkus#5006 (comment)

loicmathieu · 2023-08-25T07:39:37Z

Hi,
Still have this issue.
Does someone try to contact the Google library team to see if they can remove the offending line in their native-image.properties?

zakkak · 2023-09-07T08:10:40Z

Hi @loicmathieu , there is an upstream issue for it. See googleapis/sdk-platform-java#1344

loicmathieu · 2023-09-07T08:56:56Z

Thanks, I may try to reach out to them as we already discussed the fact that adding GraalVM commands flag in native-image.properties may have undesirable side effects.

jamesnetherton added the bug Something isn't working label Jun 22, 2023

github-actions bot added native test area/testing area/native area/google-bigquery labels Jun 22, 2023

jamesnetherton added a commit to jamesnetherton/camel-quarkus that referenced this issue Jun 22, 2023

Disable google-bigquery native tests due to apache#5006

6990e54

jamesnetherton added a commit to jamesnetherton/camel-quarkus that referenced this issue Jun 22, 2023

Disable google-bigquery native profile due to apache#5006

a937297

jamesnetherton added a commit that referenced this issue Jun 22, 2023

Disable google-bigquery native profile due to #5006

14ad98d

zakkak mentioned this issue Jun 28, 2023

Running with -H:-RunReachabilityHandlersConcurrently results in non-deterministic failures oracle/graal#6904

Closed

zakkak mentioned this issue Jun 28, 2023

Force reachability handlers to run concurrently in GraalVM 23.0 quarkusio/quarkus#34386

Closed

zhfeng added a commit to zhfeng/camel-quarkus that referenced this issue Jul 3, 2023

Revert "Disable google-bigquery native profile due to apache#5006"

9197483

This reverts commit 14ad98d.

zhfeng added a commit to zhfeng/camel-quarkus that referenced this issue Jul 3, 2023

Fix apache#5006 to add a work around for google-bigquery

584ac5c

zhfeng linked a pull request Jul 3, 2023 that will close this issue

Re-enable google-bigquery and add a work around #5041

Merged

zhfeng mentioned this issue Jul 3, 2023

Re-enable google-pubsub and google-storage and add a workaround #5033

Merged

zhfeng added a commit to zhfeng/camel-quarkus that referenced this issue Jul 4, 2023

Revert "Disable google-bigquery native profile due to apache#5006"

0c88663

This reverts commit 14ad98d.

zhfeng added a commit to zhfeng/camel-quarkus that referenced this issue Jul 4, 2023

Fix apache#5006 to add a work around for google-bigquery and apply a …

69be37e

…fix for PooledByteBufAllocatorL.java

zhfeng added a commit to zhfeng/camel-quarkus that referenced this issue Jul 4, 2023

Fix apache#5006 to add a work around for google-bigquery and apply a …

ebaa833

…fix for PooledByteBufAllocatorL.java

oubidar-Abderrahim mentioned this issue Jul 4, 2023

[GR-44056] Cannot invoke "java.util.Set.iterator()" because "triggers" is null. oracle/graal#5868

Closed

zhfeng closed this as completed in #5041 Jul 4, 2023

zhfeng added a commit that referenced this issue Jul 4, 2023

Revert "Disable google-bigquery native profile due to #5006"

706b6f7

This reverts commit 14ad98d.

zhfeng added a commit that referenced this issue Jul 4, 2023

Fix #5006 to add a work around for google-bigquery and apply a fix fo…

4499894

…r PooledByteBufAllocatorL.java

github-actions bot added this to the 3.0.0-RC1 milestone Jul 4, 2023

galderz mentioned this issue Aug 11, 2023

Native Image Build fails after upgrade to 3.2.3.Final quarkusio/quarkus#35289

Closed

ebullient added a commit to quarkiverse/quarkus-micrometer-registry that referenced this issue Aug 17, 2023

Quarkus 3.2.1 Google + Mandrel workaround

a6f80de

See apache/camel-quarkus#5006 (comment)

ebullient mentioned this issue Aug 17, 2023

Quarkus 3.2.1 Google + Mandrel workaround quarkiverse/quarkus-micrometer-registry#323

Merged

ebullient added a commit to quarkiverse/quarkus-micrometer-registry that referenced this issue Aug 17, 2023

Quarkus 3.2.1 Google + Mandrel workaround

494ea06

See apache/camel-quarkus#5006 (comment)

loicmathieu mentioned this issue Aug 25, 2023

Nullpointer Exception with native app since 2.3.0 quarkiverse/quarkus-google-cloud-services#485

Closed

This was referenced Sep 7, 2023

Native executable compiled with GraalVM fails to subscribe to Google Pub/Sub topic quarkusio/quarkus#35803

Closed

Native executable compiled with GraalVM fails to subscribe to Google Pub/Sub topic quarkiverse/quarkus-google-cloud-services#494

Closed

czp3009 mentioned this issue Sep 12, 2023

Support append BP_NATIVE_IMAGE_BUILD_ARGUMENTS to the end of command paketo-buildpacks/native-image#288

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Mandrel 23.0.0] google-bigquery integration test native compilation fails #5006

[Mandrel 23.0.0] google-bigquery integration test native compilation fails #5006

jamesnetherton commented Jun 22, 2023

jamesnetherton commented Jun 22, 2023

zhfeng commented Jun 27, 2023

jamesnetherton commented Jun 28, 2023

zhfeng commented Jun 28, 2023

jamesnetherton commented Jun 28, 2023

zakkak commented Jun 28, 2023

zakkak commented Jun 28, 2023 •

edited

Loading

jamesnetherton commented Jun 28, 2023

zakkak commented Jun 28, 2023 •

edited

Loading

zhfeng commented Jun 28, 2023

zakkak commented Jun 28, 2023

zhfeng commented Jun 29, 2023

zakkak commented Jun 29, 2023 •

edited

Loading

zhfeng commented Jun 29, 2023

zhfeng commented Jun 29, 2023

zakkak commented Jun 29, 2023

zakkak commented Jun 29, 2023 •

edited

Loading

zhfeng commented Jun 29, 2023

zhfeng commented Jun 29, 2023

zhfeng commented Jun 30, 2023

ppalaga commented Jul 3, 2023

zakkak commented Jul 3, 2023 •

edited

Loading

ppalaga commented Jul 3, 2023

loicmathieu commented Aug 25, 2023

zakkak commented Sep 7, 2023

loicmathieu commented Sep 7, 2023

[Mandrel 23.0.0] google-bigquery integration test native compilation fails #5006

[Mandrel 23.0.0] google-bigquery integration test native compilation fails #5006

Comments

jamesnetherton commented Jun 22, 2023

Bug description

jamesnetherton commented Jun 22, 2023

zhfeng commented Jun 27, 2023

jamesnetherton commented Jun 28, 2023

zhfeng commented Jun 28, 2023

jamesnetherton commented Jun 28, 2023

zakkak commented Jun 28, 2023

zakkak commented Jun 28, 2023 • edited Loading

jamesnetherton commented Jun 28, 2023

zakkak commented Jun 28, 2023 • edited Loading

zhfeng commented Jun 28, 2023

zakkak commented Jun 28, 2023

zhfeng commented Jun 29, 2023

zakkak commented Jun 29, 2023 • edited Loading

zhfeng commented Jun 29, 2023

zhfeng commented Jun 29, 2023

zakkak commented Jun 29, 2023

zakkak commented Jun 29, 2023 • edited Loading

zhfeng commented Jun 29, 2023

zhfeng commented Jun 29, 2023

zhfeng commented Jun 30, 2023

ppalaga commented Jul 3, 2023

zakkak commented Jul 3, 2023 • edited Loading

ppalaga commented Jul 3, 2023

loicmathieu commented Aug 25, 2023

zakkak commented Sep 7, 2023

loicmathieu commented Sep 7, 2023

zakkak commented Jun 28, 2023 •

edited

Loading

zakkak commented Jun 28, 2023 •

edited

Loading

zakkak commented Jun 29, 2023 •

edited

Loading

zakkak commented Jun 29, 2023 •

edited

Loading

zakkak commented Jul 3, 2023 •

edited

Loading