JDK 17 ARM64 doesnt work - Java cannot run in AWS Fargate

[carpnick]

Jenkins and plugins versions report

Environment

uname -a
Linux ip-.x.x.x.x.compute.internal 4.14.291-218.527.amzn2.aarch64 #1 SMP Fri Aug 26 09:54:28 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

What Operating System are you using (both controller, and any agents involved in the problem)?

uname -a
Linux ip-.x.x.x.x.compute.internal 4.14.291-218.527.amzn2.aarch64 #1 SMP Fri Aug 26 09:54:28 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

Reproduction steps

Try 1

1. On an amazon linux 2 machine with docker installed on ARM64 in AWS (IE t4g.medium)
2. docker pull jenkins/agent:3071.v7e9b_0dc08466-1-jdk17
3. docker run -it --entrypoint /bin/bash jenkins/agent:3071.v7e9b_0dc08466-1-jdk17
   a. Run java within the container, error happens below:
   bash: /opt/java/openjdk/bin/java: cannot execute binary file: Exec format error

Try 2

Also tried the ARM64 specific image:
docker run -it --entrypoint /bin/bash jenkins/agent:3071.v7e9b_0dc08466-1-jdk17@sha256:bc386f73b3654d7ccaa73060ee66453be35795eb48d9d670d0eaeaa38e16688e

bash: /opt/java/openjdk/bin/java: cannot execute binary file: Exec format error

Try3

If I use the preview image it works...
docker run -it --entrypoint /bin/bash jenkins/agent:latest-jdk17-preview

Command Output

jenkins@a44fdd3f66c0:~$ java
Usage: java [options] <mainclass> [args...]
           (to execute a class)
   or  java [options] -jar <jarfile> [args...]
           (to execute a jar file)
   or  java [options] -m <module>[/<mainclass>] [args...]
       java [options] --module <module>[/<mainclass>] [args...]
           (to execute the main class in a module)
   or  java [options] <sourcefile> [args]
           (to execute a single source-file program)

 Arguments following the main class, source file, -jar <jarfile>,
 -m or --module <module>/<mainclass> are passed as the arguments to
 main class.

 where options include:

    -cp <class search path of directories and zip/jar files>
    -classpath <class search path of directories and zip/jar files>
    --class-path <class search path of directories and zip/jar files>
                  A : separated list of directories, JAR archives,
                  and ZIP archives to search for class files.
    -p <module path>
    --module-path <module path>...
                  A : separated list of directories, each directory
                  is a directory of modules.
    --upgrade-module-path <module path>...
                  A : separated list of directories, each directory
                  is a directory of modules that replace upgradeable
                  modules in the runtime image
    --add-modules <module name>[,<module name>...]
                  root modules to resolve in addition to the initial module.
                  <module name> can also be ALL-DEFAULT, ALL-SYSTEM,
                  ALL-MODULE-PATH.
    --enable-native-access <module name>[,<module name>...]
                  modules that are permitted to perform restricted native operations.
                  <module name> can also be ALL-UNNAMED.
    --list-modules
                  list observable modules and exit
    -d <module name>
    --describe-module <module name>
                  describe a module and exit
    --dry-run     create VM and load main class but do not execute main method.
                  The --dry-run option may be useful for validating the
                  command-line options such as the module system configuration.
    --validate-modules
                  validate all modules and exit
                  The --validate-modules option may be useful for finding
                  conflicts and other errors with modules on the module path.
    -D<name>=<value>
                  set a system property
    -verbose:[class|module|gc|jni]
                  enable verbose output for the given subsystem
    -version      print product version to the error stream and exit
    --version     print product version to the output stream and exit
    -showversion  print product version to the error stream and continue
    --show-version
                  print product version to the output stream and continue
    --show-module-resolution
                  show module resolution output during startup
    -? -h -help
                  print this help message to the error stream
    --help        print this help message to the output stream
    -X            print help on extra options to the error stream
    --help-extra  print help on extra options to the output stream
    -ea[:<packagename>...|:<classname>]
    -enableassertions[:<packagename>...|:<classname>]
                  enable assertions with specified granularity
    -da[:<packagename>...|:<classname>]
    -disableassertions[:<packagename>...|:<classname>]
                  disable assertions with specified granularity
    -esa | -enablesystemassertions
                  enable system assertions
    -dsa | -disablesystemassertions
                  disable system assertions
    -agentlib:<libname>[=<options>]
                  load native agent library <libname>, e.g. -agentlib:jdwp
                  see also -agentlib:jdwp=help
    -agentpath:<pathname>[=<options>]
                  load native agent library by full pathname
    -javaagent:<jarpath>[=<options>]
                  load Java programming language agent, see java.lang.instrument
    -splash:<imagepath>
                  show splash screen with specified image
                  HiDPI scaled images are automatically supported and used
                  if available. The unscaled image filename, e.g. image.ext,
                  should always be passed as the argument to the -splash option.
                  The most appropriate scaled image provided will be picked up
                  automatically.
                  See the SplashScreen API documentation for more information
    @argument files
                  one or more argument files containing options
    -disable-@files
                  prevent further argument file expansion
    --enable-preview
                  allow classes to depend on preview features of this release
To specify an argument for a long option, you can use --<name>=<value> or
--<name> <value>.

Expected Results

When running java it should print the java help and load the runtime correctly:

jenkins@169274143b43:~$ java
Usage: java [options] <mainclass> [args...]
           (to execute a class)
   or  java [options] -jar <jarfile> [args...]
           (to execute a jar file)
   or  java [options] -m <module>[/<mainclass>] [args...]
       java [options] --module <module>[/<mainclass>] [args...]
           (to execute the main class in a module)
   or  java [options] <sourcefile> [args]
           (to execute a single source-file program)

 Arguments following the main class, source file, -jar <jarfile>,
 -m or --module <module>/<mainclass> are passed as the arguments to
 main class.

 where options include:

    -cp <class search path of directories and zip/jar files>
    -classpath <class search path of directories and zip/jar files>
    --class-path <class search path of directories and zip/jar files>
                  A : separated list of directories, JAR archives,
                  and ZIP archives to search for class files.
    -p <module path>
    --module-path <module path>...
                  A : separated list of directories, each directory
                  is a directory of modules.
    --upgrade-module-path <module path>...
                  A : separated list of directories, each directory
                  is a directory of modules that replace upgradeable
                  modules in the runtime image
    --add-modules <module name>[,<module name>...]
                  root modules to resolve in addition to the initial module.
                  <module name> can also be ALL-DEFAULT, ALL-SYSTEM,
                  ALL-MODULE-PATH.
    --enable-native-access <module name>[,<module name>...]
                  modules that are permitted to perform restricted native operations.
                  <module name> can also be ALL-UNNAMED.
    --list-modules
                  list observable modules and exit
    -d <module name>
    --describe-module <module name>
                  describe a module and exit
    --dry-run     create VM and load main class but do not execute main method.
                  The --dry-run option may be useful for validating the
                  command-line options such as the module system configuration.
    --validate-modules
                  validate all modules and exit
                  The --validate-modules option may be useful for finding
                  conflicts and other errors with modules on the module path.
    -D<name>=<value>
                  set a system property
    -verbose:[class|module|gc|jni]
                  enable verbose output for the given subsystem
    -version      print product version to the error stream and exit
    --version     print product version to the output stream and exit
    -showversion  print product version to the error stream and continue
    --show-version
                  print product version to the output stream and continue
    --show-module-resolution
                  show module resolution output during startup
    -? -h -help
                  print this help message to the error stream
    --help        print this help message to the output stream
    -X            print help on extra options to the error stream
    --help-extra  print help on extra options to the output stream
    -ea[:<packagename>...|:<classname>]
    -enableassertions[:<packagename>...|:<classname>]
                  enable assertions with specified granularity
    -da[:<packagename>...|:<classname>]
    -disableassertions[:<packagename>...|:<classname>]
                  disable assertions with specified granularity
    -esa | -enablesystemassertions
                  enable system assertions
    -dsa | -disablesystemassertions
                  disable system assertions
    -agentlib:<libname>[=<options>]
                  load native agent library <libname>, e.g. -agentlib:jdwp
                  see also -agentlib:jdwp=help
    -agentpath:<pathname>[=<options>]
                  load native agent library by full pathname
    -javaagent:<jarpath>[=<options>]
                  load Java programming language agent, see java.lang.instrument
    -splash:<imagepath>
                  show splash screen with specified image
                  HiDPI scaled images are automatically supported and used
                  if available. The unscaled image filename, e.g. image.ext,
                  should always be passed as the argument to the -splash option.
                  The most appropriate scaled image provided will be picked up
                  automatically.
                  See the SplashScreen API documentation for more information
    @argument files
                  one or more argument files containing options
    -disable-@files
                  prevent further argument file expansion
    --enable-preview
                  allow classes to depend on preview features of this release
To specify an argument for a long option, you can use --<name>=<value> or
--<name> <value>.

Actual Results

jenkins@b5f5793e03a8:~$ java
bash: /opt/java/openjdk/bin/java: cannot execute binary file: Exec format error

Anything else?

What is the difference between preview builds and versioned builds?

Overall I am trying to use the docker-inbound-agent on AWS Fargate ARM, but hitting issues with Java not being the correct runtime in the ARM image.

[carpnick]

Doing some initial research this line may be the culprit. Saw this issue in moby: moby/buildkit#1241

[gounthar]

Doing some initial research this line may be the culprit. Saw this issue in moby: moby/buildkit#1241

Makes sense... But we also have

To avoid "jmods: Value too large for defined data type" error,

on top of the FROM.
I guess I can try a proposal and build it on my machines, but let's hope an expert will chime in.

Thanks a lot for your report.

[dduportal]

Thanks a lot @carpnick for raising this issue and sharing the potential solution!

Looks like we forgot to remove the --platform flag for this image because https://github.com/jenkinsci/docker-agent/blob/master/11/bullseye/Dockerfile#L23 does not have the flag.

@carpnick are you ok to open a pull request, this is a nice contribution!

[gounthar]

As I suspected, we're back to the first error.

[carpnick]

I wonder if the first error is related to QEMU 32bit/64 problem : https://bugs.launchpad.net/qemu/+bug/1805913 which then gets redirected to https://gitlab.com/qemu-project/qemu/-/issues/263

Until QEMU resolves - supporting arm32 in this repo will block folks trying to use arm64. This PR it was introduced for ARM32

[gounthar]

I plead guilty.

[dduportal]

Would be worth using a multi-stage Dockerfile to fix this:

• A FROM eclipse-temurin:17.0.4.1_1-jdk-focal AS jre-build by default to fix the issue here
• A FROM --platform=linux/arm32 eclipse-temurin:17.0.4.1_1-jdk-focal AS arm32-jre-build

WDYT?

[gounthar]

Great!

[carpnick]

How would it work @dduportal ? We have 2 images and need to selectively copy based on a special platform. COPY --from doesnt support conditionals. We need a conditional somewhere that says "when this platform use image 1, when not use image 2`.

This is me thinking out loud. I have been trying this locally and can't think of a way to do this without separating the docker files.

The line that controls this assuming you get multistage working:
COPY --from=jre-build /javaruntime $JAVA_HOME

[carpnick]

Looking into docker-bake to see if we can do anything there

[dduportal]

@carpnick your analysis is correct: docker bake would be where the "condtionnal" would be by defining 2 different "targets" pointing to a Dockerfile targets each on the same Dockerfile

[gounthar]

I would love to try something, but I won't be able to address that before Wednesday.

[carpnick]

@gounthar and @dduportal - Thoughts - #310

I looked into this more and jlinks is not working for arm/v7. I tried manually and even loading in some config tools. I am not an expert on java compilation. So instead I took the approach that we just outright copy the java runtime as is in the arm32 case. I consider this the best answer of a bad situation due to the error we are running into. I think us playing with BUILDPLATFORM environment is the wrong choice as we will run into this same error with potentially a different platform. So the least risky way to go about it also increases docker image file size (at least with my knowledge).

If we think it is good - I will clean it up properly.

[timja]

Looks like a good a approach to me

[gounthar]

I haven't had a thorough look yet, but what do you mean by

jlinks is not working for arm/v7

and

the java runtime as is in the arm32 case

For me, linux/armv7 is a synonym for arm32 in the Docker world. Sorry if I muddy the water. 😊

[carpnick]

@gounthar - sorry for not being clear. The --platform has to match one of the OS/Arch of the image we are using and so the only option was to set like this. That yielded the same error result:

Jenkins Error

22:00:51  #43 [debian_jdk17 arm32-jre-build 2/2] RUN jlink   --add-modules ALL-MODULE-PATH   --strip-java-debug-attributes   --no-man-pages   --no-header-files   --compress=2   --output /javaruntime
22:00:51  #43 12.03 Error: java.nio.file.FileSystemException: /opt/java/openjdk/jmods: Value too large for defined data type
22:00:51  #43 12.03 java.nio.file.DirectoryIteratorException: java.nio.file.FileSystemException: /opt/java/openjdk/jmods: Value too large for defined data type
22:00:51  #43 12.04 	at java.base/sun.nio.fs.UnixDirectoryStream$UnixDirectoryIterator.readNextEntry(UnixDirectoryStream.java:169)
22:00:51  #43 12.04 	at java.base/sun.nio.fs.UnixDirectoryStream$UnixDirectoryIterator.hasNext(UnixDirectoryStream.java:198)
22:00:51  #43 12.04 	at java.base/jdk.internal.module.ModulePath.scanDirectory(ModulePath.java:275)
22:00:51  #43 12.04 	at java.base/jdk.internal.module.ModulePath.scan(ModulePath.java:232)
22:00:51  #43 12.04 	at java.base/jdk.internal.module.ModulePath.scanNextEntry(ModulePath.java:190)
22:00:51  #43 12.04 	at java.base/jdk.internal.module.ModulePath.find(ModulePath.java:154)
22:00:51  #43 12.04 	at jdk.jlink/jdk.tools.jlink.internal.JlinkTask.newModuleFinder(JlinkTask.java:468)
22:00:51  #43 12.04 	at jdk.jlink/jdk.tools.jlink.internal.JlinkTask.initJlinkConfig(JlinkTask.java:387)
22:00:51  #43 12.04 	at jdk.jlink/jdk.tools.jlink.internal.JlinkTask.run(JlinkTask.java:271)
22:00:51  #43 12.04 	at jdk.jlink/jdk.tools.jlink.internal.Main.run(Main.java:55)
22:00:51  #43 12.04 	at jdk.jlink/jdk.tools.jlink.internal.Main.main(Main.java:33)
22:00:51  #43 12.04 Caused by: java.nio.file.FileSystemException: /opt/java/openjdk/jmods: Value too large for defined data type
22:00:51  #43 12.04 	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
22:00:51  #43 12.04 	at java.base/sun.nio.fs.UnixException.asIOException(UnixException.java:115)
22:00:51  #43 12.04 	at java.base/sun.nio.fs.UnixDirectoryStream$UnixDirectoryIterator.readNextEntry(UnixDirectoryStream.java:168)
22:00:51  #43 12.04 	... 10 more
22:00:53  #43 ERROR: process "/bin/sh -c jlink   --add-modules ALL-MODULE-PATH   --strip-java-debug-attributes   --no-man-pages   --no-header-files   --compress=2   --output /javaruntime" did not complete successfully: exit code: 4
22:00:53  
22:00:53  #42 [archlinux_jdk11 stage-1 4/8] RUN pacman -Syu git git-lfs --noconfirm

[dduportal]

@gounthar and @dduportal - Thoughts - #310

I looked into this more and jlinks is not working for arm/v7. I tried manually and even loading in some config tools. I am not an expert on java compilation. So instead I took the approach that we just outright copy the java runtime as is in the arm32 case. I consider this the best answer of a bad situation due to the error we are running into. I think us playing with BUILDPLATFORM environment is the wrong choice as we will run into this same error with potentially a different platform. So the least risky way to go about it also increases docker image file size (at least with my knowledge).

If we think it is good - I will clean it up properly.

Make sense : the additional size is not a problem if it is only on the arm32 images.

[carpnick]

Thank you all for feedback. I think #310 is ready to go.