Docker-based build of native images fails using Minikube on macOS

[fkaltner]

Describe the bug

When running a build of a native image with the quarkus-maven-plugin NativeImageBuildRunner fails with:

[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildRunner] docker start --attach 6b91e26e205f9a6b6bb982dfe829155411eebc39fc5cee1e294895b33c9df802
[hono-adapter-amqp-vertx-quarkus-1.7.0-SNAPSHOT-runner:25]    classlist:   9.758,77 ms,  1,69 GB
[hono-adapter-amqp-vertx-quarkus-1.7.0-SNAPSHOT-runner:25]        setup:     438,75 ms,  1,69 GB
Error: Policy com.oracle.svm.core.genscavenge.CollectionPolicy cannot be instantiated.
com.oracle.svm.core.util.UserError$UserException: Policy com.oracle.svm.core.genscavenge.CollectionPolicy cannot be instantiated.
    at com.oracle.svm.core.util.UserError.abort(UserError.java:68)
    at com.oracle.svm.core.genscavenge.CollectionPolicy.instantiatePolicy(CollectionPolicy.java:68)
    at com.oracle.svm.core.genscavenge.CollectionPolicy.getInitialPolicy(CollectionPolicy.java:55)
    at com.oracle.svm.core.genscavenge.GCImpl.<init>(GCImpl.java:121)
    at com.oracle.svm.core.genscavenge.HeapImpl.<init>(HeapImpl.java:116)
    at com.oracle.svm.core.genscavenge.graal.HeapFeature.afterRegistration(HeapFeature.java:74)
    at com.oracle.svm.hosted.NativeImageGenerator.lambda$setupNativeImage$11(NativeImageGenerator.java:849)
    at com.oracle.svm.hosted.FeatureHandler.forEachFeature(FeatureHandler.java:70)
    at com.oracle.svm.hosted.NativeImageGenerator.setupNativeImage(NativeImageGenerator.java:849)
    at com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:561)
    at com.oracle.svm.hosted.NativeImageGenerator.lambda$run$0(NativeImageGenerator.java:476)
    at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1407)
    at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
    at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
    at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
    at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
    at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
Error: Image build request failed with exit status 1
ssh: Process exited with status 1

Expected behavior

The build image should be built successfully.

Actual behavior

The build of the image fails since the value of InitialCollectionPolicy parameter contains a $ which is not escaped (-H:InitialCollectionPolicy=com.oracle.svm.core.genscavenge.CollectionPolicy\$BySpaceAndTime) which can be observed in the output of NativeImageBuildRemoteContainerRunner:

[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildRemoteContainerRunner] docker create --env LANG=C quay.io/quarkus/ubi-quarkus-native-image:21.0.0-java11 -J-Dsun.nio.ch.maxUpdateArraySize=100 -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-Dvertx.logger-delegate-factory-class-name=io.quarkus.vertx.core.runtime.VertxLogDelegateFactory -J-Dvertx.disableDnsResolver=true -J-Dio.netty.leakDetection.level=DISABLED -J-Dio.netty.allocator.maxOrder=1 -J-Duser.language=en -J-Duser.country=DE -J-Dfile.encoding=UTF-8 --initialize-at-run-time=io.netty.internal.tcnative.SSL -H:ResourceConfigurationFiles=resources-config.json -H:ReflectionConfigurationFiles=reflection-config.json --initialize-at-build-time= -H:InitialCollectionPolicy=com.oracle.svm.core.genscavenge.CollectionPolicy\$BySpaceAndTime -H:+JNI -H:+AllowFoldMethods -jar hono-adapter-amqp-vertx-quarkus-1.7.0-SNAPSHOT-runner.jar -H:FallbackThreshold=0 -H:+ReportExceptionStackTraces -H:-AddAllCharsets -H:EnableURLProtocols=http,https --enable-all-security-services --no-server -H:-UseServiceLoaderFeature -H:+StackTrace hono-adapter-amqp-vertx-quarkus-1.7.0-SNAPSHOT-runner

This leads to the following Run Command in the Docker container which is used for the build of the native image:

"Cmd": [
               ...
                "-H:InitialCollectionPolicy=com.oracle.svm.core.genscavenge.CollectionPolicy",
                ...
            ],

Obviously the CollectionPolicy cannot be instantiated when the container is run because the reference to its factory method is missing.

To Reproduce

Here's a link to a branch of Eclipse Hono to reproduce the problem: https://github.com/bosch-io/hono/tree/quarkus_native_image_minikube_macos_reproducer

Steps to reproduce the behavior:

1. Setup Minikube on macOS using the hyperkit driver. Mount the hosts /Usersdirectory into Minikube by defining a NFS share on the host in /etc/exports (/Users -mapall=<yourname> -network 192.168.64.0 -mask 255.255.255.0) and start Minikube with --nfs-share "/Users" and --nfs-shares-root “/“
2. The "docker" executable has to be a wrapper script on your local machine (see: Docker-based build of native images fails using Minikube on macOS #16040 (comment))
3. Clone and checkout the aforementioned branch of Eclipse Hono
4. Run eval $(minikube docker-env)
5. Run mvn clean install -pl '!site' -Pbuild-docker-image,metrics-prometheus,jaeger,build-native-image

Environment (please complete the following information):

Output of uname -a or ver

Darwin <>-MacBook-Pro.local 20.3.0 Darwin Kernel Version 20.3.0: Thu Jan 21 00:07:06 PST 2021; root:xnu-7195.81.3~1/RELEASE_X86_64 x86_64 i386 MacBookPro16,1 Darwin

Output of java -version

Picked up _JAVA_OPTIONS: -Dhttp.nonProxyHosts=localhost|127.*|192.168.*
openjdk version "15.0.1" 2020-10-20
OpenJDK Runtime Environment (build 15.0.1+9)
OpenJDK 64-Bit Server VM (build 15.0.1+9, mixed mode, sharing)

GraalVM version (if different from Java)

[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] Running Quarkus native-image plugin on GraalVM Version 21.0.0 (Java Version 11.0.10+8-jvmci-21.0-b06)

Quarkus version or git rev

1.13.0.CR1

Build tool (ie. output of mvnw --version or gradlew --version)

Picked up _JAVA_OPTIONS: -Dhttp.nonProxyHosts=localhost|127.*|192.168.*
Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
Maven home: /usr/local/Cellar/maven/3.6.3_1/libexec
Java version: 11.0.9, vendor: Oracle Corporation, runtime: /usr/local/Cellar/openjdk@11/11.0.9/libexec/openjdk.jdk/Contents/Home
Default locale: en_DE, platform encoding: UTF-8
OS name: "mac os x", version: "10.16", arch: "x86_64", family: "mac"

Additional context

minikube version: v1.17.1
commit: 043bdca07e54ab6e4fc0457e3064048f34133d7e

There's also another PR with the same problem which has been closed due to inactivity: #2435 (review)

[quarkus-bot]

/cc @geoand

[geoand]

@jonathan-meier this look like something that must be fixed as a result of #14635, no?

[jonathan-meier]

From what I can tell, it doesn't seem to be a bug on Quarkus' side, but rather an issue with minikube. As @dmlloyd already stated in #2435 we're using ProcessBuilder which does not do any string interpolation since there is no shell involved. However, it seems that minikube internally works with wrapper shell scripts in order to be accessible via the docker command which in all likelihood do the string interpolation instead of simply forwarding the command faithfully (effectively removing $BySpaceAndTime, since the shell variable $BySpaceAndTime does not exist).

Note also that the escape character in the log message is actually only inserted for the log message itself, meaning the actual command contains the dollar character unescaped:

quarkus/core/deployment/src/main/java/io/quarkus/deployment/pkg/steps/NativeImageBuildRemoteContainerRunner.java

Line 29 in 9e8cf8c

log.info(String.join(" ", createContainerCommand).replace("$", "\\$"));

Unfortunately, I neither have any experience with minikube nor a setup to verify my claims.

[jonathan-meier]

A possible workaround might actually be to inject the shell variable BySpaceAndTime=$BySpaceAndTime into the environment where the string interpolation happens 😃

[geoand]

Sneaky!

[jonathan-meier]

@fkaltner I wasn't able to reproduce it. The containerized native image build worked fine for me in the minikube cluster on macOS ($BySpaceAndTime stayed attached to the CollectionPolicy class in the command, as expected), though I'm not entirely sure whether I did the same thing as you do.

What is the relevance of the NFS share arguments for minikube start? I don't see where in the reproduction we need to access the /Users directory from within a container, so I skipped these arguments.

Moreover, I did eval $(minikube docker-env) before the mvn build command in order to actually use the minikube cluster. I suppose you just forgot to mention that explicitly, right?

[fkaltner]

@jonathan-meier Yes, you are right, eval $(minikube docker-env) is required before the mvn build command. I'll update the steps to reproduce.

Since it worked on your machine, I carefully re-inspected my local Minikube setup looking for any wrapper scripts for the "docker" command. Actually it turned out I created one (named "docker"), since Minikube does not ship with a local "docker" command:

#!/bin/sh
minikube ssh -- docker $@

I guess the $@ will also do string interpolation.

The solution was to delete that wrapper script from my local "/usr/local/bin" directory and install docker via brew install docker.

Thank you for your help! 👍

 
 
 

For the record (note that I think this only applies if one uses a wrapper script for "docker" with Minikube):

The relevance of the NFS share arguments is the docker cp command which is used to copy the "runner.jar" amongst other things:

[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildRunner] docker cp /Users/fkaltner/git/hono/adapters/amqp-vertx-quarkus/target/hono-adapter-amqp-vertx-quarkus-1.7.0-SNAPSHOT-native-image-source-jar/. de22faae4f124c9e3c3afbc96307c189e5063273b1e8e58cc3c8090879d3a8ee:/project

As you correctly noticed docker itself runs in a VM in Minikube. This "docker cp" command is consequently run in the VM, so the path starting with "/Users/..." needs to be present in the VM. For VirtualBox that mount is done automatically but for Hyperkit one has to take care of it manually (see section "Driver mounts" in https://minikube.sigs.k8s.io/docs/handbook/mount/).