Copy sources from remote container when quarkus.native.debug.enabled=true #15288

zakkak · 2021-02-24T01:05:05Z

When using -Dquarkus.native.remote-container-build=true in combination with -Dquarkus.native.debug.enabled=true native image generation fails with:

[ERROR] Failed to execute goal io.quarkus:quarkus-maven-plugin:999-SNAPSHOT:build (default) on project quarkus-integration-test-main: Failed to build quarkus application: io.quarkus.builder.BuildException: Build failure: Build failed due to errors
[ERROR]         [error]: Build step io.quarkus.deployment.pkg.steps.NativeImageBuildStep#build threw an exception: java.lang.RuntimeException: Failed to build native image
[ERROR]         at io.quarkus.deployment.pkg.steps.NativeImageBuildStep.build(NativeImageBuildStep.java:209)
[ERROR]         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[ERROR]         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[ERROR]         at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[ERROR]         at java.base/java.lang.reflect.Method.invoke(Method.java:566)
[ERROR]         at io.quarkus.deployment.ExtensionLoader$2.execute(ExtensionLoader.java:920)
[ERROR]         at io.quarkus.builder.BuildContext.run(BuildContext.java:277)
[ERROR]         at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2415)
[ERROR]         at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1452)
[ERROR]         at java.base/java.lang.Thread.run(Thread.java:834)
[ERROR]         at org.jboss.threads.JBossThread.run(JBossThread.java:501)
[ERROR] Caused by: java.nio.file.NoSuchFileException: /home/zakkak/code/quarkus/integration-tests/main/target/quarkus-integration-test-main-999-SNAPSHOT-native-image-source-jar/sources
[ERROR]         at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
[ERROR]         at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
[ERROR]         at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
[ERROR]         at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
[ERROR]         at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149)
[ERROR]         at java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
[ERROR]         at java.base/java.nio.file.Files.readAttributes(Files.java:1764)
[ERROR]         at java.base/java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:225)
[ERROR]         at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
[ERROR]         at java.base/java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:322)
[ERROR]         at java.base/java.nio.file.Files.walkFileTree(Files.java:2717)
[ERROR]         at io.quarkus.bootstrap.util.IoUtils.copy(IoUtils.java:126)
[ERROR]         at io.quarkus.deployment.pkg.steps.NativeImageBuildStep.build(NativeImageBuildStep.java:191)
[ERROR]         ... 10 more

This PR fixes this by copying the sources from the container back to the host.

zakkak · 2021-02-24T01:06:05Z

\cc @jonathan-meier

core/deployment/src/main/java/io/quarkus/deployment/pkg/steps/NativeImageBuildRunner.java

jonathan-meier

Looks good to me, though I think we already missed copying some more artefacts, e.g. the dashboard dump:

quarkus/core/deployment/src/main/java/io/quarkus/deployment/pkg/steps/NativeImageBuildStep.java

Lines 796 to 799 in faa48d7

    
           if (nativeConfig.enableDashboardDump) { 
        
               nativeImageArgs.add("-H:DashboardDump=" + outputTargetBuildItem.getBaseName() + "_dashboard.dump"); 
        
               nativeImageArgs.add("-H:+DashboardAll"); 
        
           }

and the native sources for the native sources build:

quarkus/core/deployment/src/main/java/io/quarkus/deployment/pkg/steps/NativeImageBuildStep.java

Line 83 in faa48d7

    
           outputDir = buildSystemTargetBuildItem.getOutputDirectory().resolve("native-sources");

So it might make sense to assemble a list of output artefacts in NativeImageBuildStep and hand that list over to the NativeImageBuildRunner.

Also, thanks for catching the issue with the user. I missed that because in my tests the local user had the same UID as the quarkus user in the container (UID 1001). Sorry about that!

...ent/src/main/java/io/quarkus/deployment/pkg/steps/NativeImageBuildRemoteContainerRunner.java

…true

Using them results in files being copied back to host to be owned by the guest user instead of the host user. e.g. $ podman create --name temp --user 1000:1000 --userns=keep-id -it \ quay.io/quarkus/ubi-quarkus-native-image:21.0.0-java11 $ podman cp temp:/opt/graalvm/bin/native-image remote-native-image $ ls -la remote-native-image -rwxr-xr-x. 1 100000 100000 14641161 Feb 14 03:28 remote-native-image* $ id -u 1000

core/deployment/src/main/java/io/quarkus/deployment/pkg/steps/NativeImageBuildLocalRunner.java

geoand · 2021-03-02T06:26:39Z

core/deployment/src/main/java/io/quarkus/deployment/pkg/steps/NativeImageBuildRunner.java

+            if (workingDirectory != null) {
+                processBuilder.directory(workingDirectory);
+            }
+            process = processBuilder.start();


Looks like you are you are not redirecting the output streams like ProcessUtil#launchProcess was doing

Indeed, but that's how things where before (except for cleanupServer).

runCommand is used to run native-image --shutdown-server, docker rm container-name and docker cp src dest. IMO it's OK to not redirect output for these commands (as discussed in https://github.com/quarkusio/quarkus/pull/15288/files#r582745324). WDYT?

Thanks George

Thanks for fixing this

jonathan-meier · 2021-03-02T13:34:50Z

I just realized that #14635 has been backported to 1.12.1. Would be nice if this PR could be backported as well, mainly because of the docker user fix on Linux.

geoand · 2021-03-02T13:38:56Z

I am pretty sure it would clash as #15233 is not backported, so we likely won't backport this unless it's necessary.
In that case @zakkak would likely need to do a new PR of this against the 1.12 branch

jonathan-meier · 2021-03-02T13:51:23Z

Oh, you are right. Even though the conflicting changes in NativeImageBuildStep are tiny, it can't be rebased cleanly.

jonathan-meier · 2021-03-02T21:22:54Z

Sorry to bring this up once more, it is actually enough to just cherry-pick the second commit of this PR (5d4f39d) onto 1.12. Without this fix, the remote container build is hardly usable on 1.12 as it is.

I opened PR #15420 against 1.12 with this change. If this is undesirable or goes against your backporting protocols, feel free to just close it!

geoand · 2021-03-02T21:25:01Z

It's up to @gsmet to decide.

quarkus-bot bot added the area/core label Feb 24, 2021

geoand reviewed Feb 24, 2021

View reviewed changes

core/deployment/src/main/java/io/quarkus/deployment/pkg/steps/NativeImageBuildRunner.java Outdated Show resolved Hide resolved

jonathan-meier reviewed Feb 24, 2021

View reviewed changes

...ent/src/main/java/io/quarkus/deployment/pkg/steps/NativeImageBuildRemoteContainerRunner.java Outdated Show resolved Hide resolved

zakkak force-pushed the copy-sources-in-containerized-build branch from 7319acc to 7975a6b Compare February 24, 2021 12:11

zakkak added 3 commits February 24, 2021 14:37

Copy sources from remote container when quarkus.native.debug.enabled=…

2eb61b1

…true

Improve logging in native image builder

ea2159f

zakkak force-pushed the copy-sources-in-containerized-build branch 2 times, most recently from b2904f0 to 63fec3d Compare February 24, 2021 13:56

geoand reviewed Feb 24, 2021

View reviewed changes

core/deployment/src/main/java/io/quarkus/deployment/pkg/steps/NativeImageBuildLocalRunner.java Show resolved Hide resolved

NativeImageBuildRunner: Improve error handling

bff39cf

zakkak force-pushed the copy-sources-in-containerized-build branch from 63fec3d to bff39cf Compare February 25, 2021 11:13

zakkak requested a review from geoand March 1, 2021 23:31

geoand reviewed Mar 2, 2021

View reviewed changes

geoand approved these changes Mar 2, 2021

View reviewed changes

geoand merged commit 19e6436 into quarkusio:master Mar 2, 2021

quarkus-bot bot added this to the 1.13 - master milestone Mar 2, 2021

zakkak deleted the copy-sources-in-containerized-build branch March 2, 2021 07:22

jonathan-meier mentioned this pull request Mar 2, 2021

Fix file permission issue for container builds on remote daemons #15420

Merged

Karm mentioned this pull request Dec 7, 2021

Custom native container runtime options are overwritten #17223

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Copy sources from remote container when quarkus.native.debug.enabled=true #15288

Copy sources from remote container when quarkus.native.debug.enabled=true #15288

zakkak commented Feb 24, 2021

zakkak commented Feb 24, 2021

jonathan-meier left a comment

geoand Mar 2, 2021

zakkak Mar 2, 2021

geoand Mar 2, 2021

zakkak Mar 2, 2021

geoand Mar 2, 2021

jonathan-meier commented Mar 2, 2021

geoand commented Mar 2, 2021

jonathan-meier commented Mar 2, 2021

jonathan-meier commented Mar 2, 2021

geoand commented Mar 2, 2021

	if (nativeConfig.enableDashboardDump) {
	nativeImageArgs.add("-H:DashboardDump=" + outputTargetBuildItem.getBaseName() + "_dashboard.dump");
	nativeImageArgs.add("-H:+DashboardAll");
	}

Copy sources from remote container when quarkus.native.debug.enabled=true #15288

Copy sources from remote container when quarkus.native.debug.enabled=true #15288

Conversation

zakkak commented Feb 24, 2021

zakkak commented Feb 24, 2021

jonathan-meier left a comment

Choose a reason for hiding this comment

geoand Mar 2, 2021

Choose a reason for hiding this comment

zakkak Mar 2, 2021

Choose a reason for hiding this comment

geoand Mar 2, 2021

Choose a reason for hiding this comment

zakkak Mar 2, 2021

Choose a reason for hiding this comment

geoand Mar 2, 2021

Choose a reason for hiding this comment

jonathan-meier commented Mar 2, 2021

geoand commented Mar 2, 2021

jonathan-meier commented Mar 2, 2021

jonathan-meier commented Mar 2, 2021

geoand commented Mar 2, 2021