Add support for arm64

[MarcelCoding]

Add support to use the buildpacks (and the corresponding docker images) on arm64.

https://app.slack.com/client/TUKS9UTJQ/CULAS8ACD/thread/CULAS8ACD-1615579116.000900

Hello, there’s a similar use case with the macs with the new Apple Silicon (M1 chip). The Spring Native builds fail as it needs binaries of AMD arch to compile. The following link might shed some light on the issue. Thanks. spring-projects/spring-boot#25619 (comment)

I am using spring boot, with the new beta native compilation for the graalvm. The graalvm is able to compile amd64 and arm64 binaries. I would like to use this on some raspberry pi's, because they have already a heavy load.

[martyspiewak]

Thanks for bringing this up!

This would require a large effort across all buildpacks and stacks so there is a quite a bit of work required to do this but we are going to look into it and see what would be involved!

[tigerinus]

We've been using AWS Graviton arm based EC2 instances for a while. Recently looking into replacing traditional docker build with paketo buildpack for our spring-boot applications. Waiting patiently for this to happen. Thanks!

[sophiewigmore]

Hey @tigerinus thanks for letting us know about your interest in this!

The upstream CNB project responsible for the buildpack lifecycle and pack platform need to implement something compatible with ARM before we can get this working in Paketo. There is an open issue and discussion around this here: buildpacks/lifecycle#435. There's also an arm64-specific Slack channel in CNB Slack if you want to connect with folks directly: https://buildpacks.slack.com/archives/CTXA5DNRG

[goafabric]

@sophiewigmore
Any news on this one ? The upstream issue seems to be fixed.
I would really like to see this feature to use it within Spring Boot Native Binaries

[goafabric]

Hello anybody in here ?

[ryanmoran]

Hi @goafabric, we are still waiting for the lifecycle upstream to be released with arm64 support. As you can see in buildpacks/lifecycle#435 (comment), the release is not out yet. Once that happens, we can take the steps required to setup support for arm64 including testing infrastructure.

[goafabric]

@ryanmoran ok thank you

[darylrobbins]

Lifecycle 0.12.0-rc.1 was just released with arm64 support

[darylrobbins]

Lifecycle v0.12.0 is now officially released with initial arm64 support.

[sophiewigmore]

Hey @darylrobbins thanks for letting us know. While we're aware that is is now available in the lifecycle, it's not a simple change for us to integrate with. Support for arm64 will involve a lot of moving pieces involving building out a set of new stacks, setting up brand new automation to test/ship everything, and ensuring that everything works with all of the Paketo buildpacks and other dependencies.

It's not something we have fully scoped out or prioritized at the current moment, so it might be a while until this is something supported officially in Paketo. When the time comes that the core development team is able to prioritize this we will update. Until then, contributions are welcome

[wind57]

@sophiewigmore

It's not something we have fully scoped out or prioritized at the current moment

This is sad. spring-boot maven plugin uses buildpacks under the hood and anyone with an M1 is going to find out that this is not supported. I am kind of amazed this issue has not seen more attention from such people. We might have to return, literally, 20 Macs, because of this missing thing. :(

Are there any plans on the roadmap?

[darylrobbins]

I'm in the same boat trying to use the Spring Boot gradle plugin, especially now that they stopped shipping Macs with x86 processors. Build packs are definitely a better way to build a docker image for an application, but you could also resort to using the standard docker plugin and a docker file to make it work for the time being.

[wind57]

We have processes that are so heavily involved (pipelines, processes, etc etc) that such changes are much too drastic, in order to switch to newer macs... The bigger problem is that if we switch to JIB (not sure if that works, but in theory), we will probably never return to spring for this, which is a pity, again.

[darylrobbins]

I am just another user of the paketo buildpacks through the Gradle Spring Boot plugin, so all I can do is offer suggestions. Technically speaking, I tried a couple buildpacks and they seemed to build fine on arm64 (using my PR to the stacks repo). So, if you're heavily dependent on the buildpacks, you could build your own versions for arm64 and use those. However, that would require a fair amount of work to do -- so it will really depend how tied you are to the process.

[tigerinus]

Business wise, the decision to de-prioritize this is basically giving up the 30% of the cloud market which runs on ARM64...not to mention that this market share is still the growing fast.

[goafabric]

To me this depriorization is also very mind boggling.
For Quarkus the support building images with standards like Dockerfile or JIB.
And while they also do not support ARM64 support right of the box, it can then be fairly easy introduced
by providing your own image builder with GraalVM Standards
https://dev.to/nevernull/building-a-native-java-application-for-arm64-with-quarkus-2n9m

[sophiewigmore]

We appreciate hearing all of your feedback on this, the need for this feature has definitely been heard. We're looking into what will be needed enable contributions and unblock work on this (like @darylrobbins PR). I plan to surface this at next week's (Nov 30th) Paketo working group as well, so we can figure out a plan forward.

[dmikusa]

I don't have a new mac, but I've been told that if you start a container using rosetta then buildpacks will work. It just defaults to using arm64, so by default, it doesn't work. Many things, not just buildpacks, are still catching up to this change.

If you are having trouble getting buildpacks to run on an MBP M1 under Rosetta/emulation, please jump in our Slack and we can help troubleshoot. https://paketobuildpacks.slack.com

The key, as I understand it, to getting this to work is that when you run docker run -it ubuntu:latest uname -a you see x86_64, not arm64. If you still see arm64, then you don't have your terminal set up to run under Rosetta/Emulation. If you see x86_64 in the output, then buildpacks should work both with pack or through Maven/Gradle & Spring Boot. Again, if you're having trouble, feel free to reach out in Slack and we can help get you pointed in the right direction.

[wind57]

@dmikusa-pivotal you're right, as I said here for example, docker run works just fine.

The problem is containerd, that does not seem to like that, because of the incorrect layers. I only noticed that since I was trying to test locally a defect against spring-cloud-kubernetes that requires an image build, by default, with buildpacks, that must be loaded into kind

[Superv0n]

Business wise, the decision to de-prioritize this is basically giving up the 30% of the cloud market which runs on ARM64...not to mention that this market share is still the growing fast.

@tigerinus This is an interesting stat (30% cloud market running on ARM64). I'd love to hear more. Would you be able to share where your research found this information please? Unless you are primarily talking about the market share that AWS has with their Gravitron2 architecture.

[dmikusa]

@wind57

@dmikusa-pivotal you're right, as I said here for example, docker run works just fine.

The problem is containerd, that does not seem to like that, because of the incorrect layers. I only noticed that since I was trying to test locally a defect against spring-cloud-kubernetes that requires an image build, by default, with buildpacks, that must be loaded into kind

Can you expand on this? What doesn't it like?

You should be able to build using buildpacks and run using Docker on your MBP with this method. Taking those images and running them elsewhere should be possible on other x86_64 systems. You wouldn't be able to build this way and run on an arm64 system.

[wind57]

I removed my comment since it has no relevance now and just muddies the discussion.

You wouldn't be able to build this way and run on an arm64 system

and that is exactly the problem, no? I want to build an image, load it in the cluster (that runs on M1) and be happy. Currently, as the linked issue explains, this is not possible.

Or let's put it even simpler.

• I build the image locally (I am on M1) with spring maven plugin, run it with docker run .... It works.
• I take the same image, load it into kind, try to use the application, it does not work, thus this issue

Do I make sense now?

[dmikusa]

@wind57

I removed my comment since it has no relevance now and just muddies the discussion.

You wouldn't be able to build this way and run on an arm64 system

and that is exactly the problem, no? I want to build an image, load it in the cluster (that runs on M1) and be happy. Currently, as the linked issue explains, this is not possible.

If you want to run it on an M1, you always have the option to run under Rosetta emulation, so you don't need arm64 support.

Or let's put it even simpler.

• I build the image locally (I am on M1) with spring maven plugin, run it with docker run .... It works.
• I take the same image, load it into kind, try to use the application, it does not work, thus this issue

If you build under Rosetta emulation, it should build an x86_64 container. You should be able to run that elsewhere (on other x86_64 systems or also under Rosetta), if that's not the case and the image doesn't work on another x86_64 system, then I would suggest opening a bug against https://github.com/buildpacks/lifecycle/issues (or possibly pack, but I think lifecycle is right given my understanding of things). The lifecycle is that takes everything the buildpacks do and wraps it up into an image. If that image isn't generated in a compatible way, we'd want to talk with the lifecycle team to understand what is happening.

[wind57]

imho, the confusion comes from here :

If you want to run it on an M1, you always have the option to run under Rosetta emulation, so you don't need arm64 support.

I am not sure what you mean, but that image ends up in kind, that uses containerd - so that seems to be the problem to me. I am by far not an expert, but this is what Ben hinted in the issue I opened.

But may be main confusion comes from the fact that to my second point:

I take the same image, load it into kind, try to use the application, it does not work...

I use kind local too, so both actions (docker run and load into kind) happen on the same M1.

wdyt, this clears it up a little?

[dmikusa]

imho, the confusion comes from here :

If you want to run it on an M1, you always have the option to run under Rosetta emulation, so you don't need arm64 support.

I am not sure what you mean, but that image ends up in kind, that uses containerd - so that seems to be the problem to me. I am by far not an expert, but this is what Ben hinted in the issue I opened.

Sorry, I can't speak to your particular case. I worked with a colleague that had an M1 and we were able to get buildpacks working by using a Rosetta terminal and Docker Desktop (this makes an x86_64 container, so as far as pack, lifecycle & buildpacks are concerned, it's all business as usual). It sounds like you have a different setup, and I can't really speak to that setup and I don't have an M1 to try that setup, sorry. There could be additional challenges.

But may be main confusion comes from the fact that to my second point:

I take the same image, load it into kind, try to use the application, it does not work...

I use kind local too, so both actions (docker run and load into kind) happen on the same M1.

If you run docker run -it ubuntu:latest uname -a what do you see? Does it give you an x86_64 container or arm64 with your setup? If it say arm64, that's not going to work. It would be effectively trying to run something that's built for x86_64 in arm64.

If you can get an x86_64 container, things should build & run. To the tools & buildpacks, it's just business as usual in an x86_64 world. If run fails for some reason, then try docker run -it --entrypoint bash <your-image> uname -a. That should run uname in your container instead of the default process. Again, double-check it's running in an x86_64 container. If it is running in an x86_64 container and still not working, open a lifecycle issue & include details of the issue (as much logging as you can & error messages) as well as a sample image if possible. Feel free to tag me on that as well. We can look more into why it's failing.

[wind57]

This is what I see from docker run -it ubuntu:latest uname -a

Linux 2f26cb4b3756 5.10.47-linuxkit #1 SMP PREEMPT Sat Jul 3 21:50:16 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux

It seems we are spamming this thread quit a bit, btw. I will try to join that slack channel instead.

[dmikusa]

Thanks, there are #arm channels on both buildpacks and Paketo slack. Talk there soon.

[gmuth]

docker (for m1) can very well run x86 images. it depends on the platforms supported by the image. Then you can choose:

$ docker run --platform=linux/amd64 -it ubuntu:latest uname -a
Linux dc70974eb416 5.10.47-linuxkit #1 SMP PREEMPT Sat Jul 3 21:50:16 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

• Dockerfile? works (not using stacks)

Manually writing a Dockerfile works because I am in control of the base image and the platform options. So I can choose the target platform (multiplatform images is another story but also possible)

# base image supports arm64 and amd64 and ...
FROM openjdk:11
# FROM --platform=linux/amd64 openjdk:11
COPY target/springboot-app.jar /app.jar
EXPOSE 8080
ENTRYPOINT ["java", "-Djava.security.egd=file:/dev/./urandom", "-jar", "/app.jar"]

• spring-boot:build-image? fails (not a stacks issue)

I have issues with the tools that should build the spring-boot-image (using an arm64-jdk)

$ mvnw spring-boot:build-image
[ERROR] Failed to execute goal org.springframework.boot:spring-boot-maven-plugin:2.4.3:build-image (default-cli) on project springworx:
Execution default-cli of goal org.springframework.boot:spring-boot-maven-plugin:2.4.3:build-image failed:
An API incompatibility was encountered while executing org.springframework.boot:spring-boot-maven-plugin:2.4.3:build-image:
java.lang.UnsatisfiedLinkError: /Users/gmuth/Library/Caches/JNA/temp/jna5109151359629377006.tmp: 
dlopen(/Users/gmuth/Library/Caches/JNA/temp/jna5109151359629377006.tmp, 0x0001): 
tried: '/Users/gmuth/Library/Caches/JNA/temp/jna5109151359629377006.tmp' (fat file, but missing compatible architecture (have 'i386,x86_64', need 'arm64e')), '/usr/lib/jna5109151359629377006.tmp' (no such file)

scottfrederick from the spring boot teams says:

"There is no workaround in Spring Boot, as the problem is not in this project. A fix or workaround will have to come from the Paketo buildpacks team, Docker Engine, or QEMU."

But this doesn't seem right, because I get a JNA error which is related to java.

• Is there a work around? (not using stacks)

Josh Long's tweet looks promising...

[goafabric]

Honestly .. i think there is currently a lot of confusion going on here.
Just for clarifaction:

• It is of yourse possible to RUN Spring Native X86 Images in Docker on Apple Silicon
• And for a while it even seems to be possible to use the Packeto X86 Builder inside Docker on Apple Silicon

BUT:
In both cases Docker will just patch the running Container during Startup with QUEMU.
This has NOTHING to do with Rosetta .. Rosetta is never ever used here ...

The Problem:
Is .. that running these Native X86 Images via Quemu leads to much inferior Performance .. both for CPU and Memory Load
And for the Paketo build .. while this works in theory, it is about 10x times ! slower than when using an ARM Variant
(i know this, because it works perfectly fine with QUarkus on my Apple Silicon)
And this just makes it toally unusable .... which is now wonder because quemu will have to emulate the complete GRAALVM Build Process ....

@gmuth
You could make it work, the JNA error might be because u are using am ARM JDK like Zulu ?
If you replace that with an X86 AdoptJDK .. it might work (it worked for me at least once)
But see me comment before .. this will yield to a build process that's 10x slower .. which renders this nearly unusable

[goafabric]

One small correction to my last post:
Docker on M1 acutally used Rosetta for some smaller things like the dashboard.
But afaik, the Core Application was always Rosetta free.
Well as of today, Rosetta has been removed entirely with Docker 4.3.0

[gmuth]

I agree about the confusion. We need to carefully distinguish which component we talk about.
The workflow of building an image involves quite a few components.

1. is the toolchain available nativly?
   as of the spring-boot-maven-plugin (including it's dependencies) no, should be addressed their mainainers
2. is the outcoming artifact (the docker image) what we expect?
   NO. I expect to get an arm64 image as a result.

[dmikusa]

Thanks for clarifying @goafabric

[dmikusa]

Hi all. For those that are interested in Java ARM64 support.

I have preliminary support for arm64 and Java available. This will allow you to with relative ease package your own ARM64 stack, buildpacks, and builder that are capable of running Java and Java Native Image applications. Please see https://github.com/dmikusa-pivotal/paketo-arm64 for more details.

This is the first step in arm64 support, not a final solution. I'd very much like folks to try it out and give feedback on what works/doesn't work for you. I know top of everyone's mind is having images (stacks, buildpacks & builders) published from the Paketo project. That's our end goal and this is the first step on that journey.

For more details on ARM64 support, you may also want to check out our #arm channel on the Paketo Slack.

Thanks!

[klopfdreh]

I just entered the discussion and sorry if I missed something. 😃

I agree about the confusion. We need to carefully distinguish which component we talk about. The workflow of building an image involves quite a few components.

1. is the toolchain available nativly?
   as of the spring-boot-maven-plugin (including it's dependencies) no, should be addressed their mainainers
2. is the outcoming artifact (the docker image) what we expect?
   NO. I expect to get an arm64 image as a result.

Indeed - I would also like to see docker images build for arm64 as output. As of spring native I saw 3 images of packeto are used while following the spring guides (I added runImage to make it more verbose):

• gcr.io/paketo-buildpacks/java-native-image
• paketobuildpacks/builder
• paketobuildpacks/run

You are able to configure the images used in the maven pom.xml:

            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <configuration>
                    <image>
                        <builder>paketobuildpacks/builder:tiny</builder>
                        <runImage>paketobuildpacks/run:latest</runImage>
                        <env>
                            <BP_NATIVE_IMAGE>true</BP_NATIVE_IMAGE>
                        </env>
                        <buildpacks>
                            <buildpack>gcr.io/paketo-buildpacks/java-native-image:7.8.1</buildpack>
                        </buildpacks>
                    </image>
                </configuration>
            </plugin>

So if I understand it correctly all tools used in those containers need to be checked for their compatibility with arm64 and the images need to be build accordingly and registered under the same tag with arm64 architecture.

This would lead to an image that is created for the architecture the maven build was executed on. (In case of spring, maven, java).

Is this assumption right?

@dmikusa-pivotal thanks for the guide 👍

[dmikusa]

@klopfdreh - Yes, basically to get it to work on ARM64, everything from the bottom OS layer up through everything the buildpacks install (i.e. the JVM, etc..) all needs to be compatible with ARM64.

To that point, the instructions posted in the previous comment walk you through everything that's necessary to build a fully ARM64 set of images. At this point in time, it's required for you to build your own images. It's early stages. Please try though and report any feedback.

The current plan is to eventually post official public ARM64 images. There is some upstream work required with buildpack tools to make that happen, so we the Paketo project can't give a timeline. We'll keep this thread up-to-date as more information is available.

[agracey]

Thanks for the update! If we can get you access to an IBM Z build machine, would you be able to replicate the port for s390x as well? (Assuming we can get upstream to make the same port for the tooling)

[dmikusa]

@agracey This would probably be better as a separate discussion. Can you open up a new issue?