Cross Build Images for ARM

[RichiCoder1]

Summary

Currently ArgoCD supports building arm64 images but doesn't publish them due to issues with cross-building with normal docker and GitHub Actions (AFAIK). There's some recent advancements that would enable this with GitHub Actions.

Motivation

Running ArgoCD on arm64 via registry images.

Proposal

Long story short, there's some excellent actions that will be landing shortly: docker/build-push-action#92 (comment)

They, amgonst other things, make it easier to cross build using QEMU (at the price of emulation overhead): https://github.com/crazy-max/test-docker-action/blob/master/.github/workflows/full.yml#L59

Figured I'd create this ticket after doing some research and noting that the limitation mostly seemed to be docker/CI related.

[ellisvalentiner]

@RichiCoder1 are you looking for help with this issue?

[RichiCoder1]

Hmmm. I could certainly take a swing, but the docker build is deeply baked into ArgoCD's make so I'd likely need guidance (or a long weekend which I won't have for a little bit haha)

[alinbalutoiu]

Another faster option would be to use a CI that supports the ARM systems.
For example, Drone CI (free for open source) includes ARM servers which allows you to easily build the image. The servers are also quite powerful.

I'm currently using that one to run and check that the ARM images work successfully. On top of the release branches there is this commit that does the build for ARM images, the build history is here.

[agustinvinao]

Hi @alinbalutoiu, Im trying to deploy argo on a Pis cluster. I was checking the images you'd build. Are those images accessible or did you put it in a private registry?

It would be great to have ArgoCD images for Arm64 available.

thanks

[ellisvalentiner]

@agustinvinao I believe these are the builds

[zeph]

@alinbalutoiu perfectly working! thanks mate ...I'm running a microk8s cluster on RP4+ (8GB)

[slmingol]

Above worked w/ my k3s cluster on RPi4+ as well. Thanks for the images!

[DWSR]

@RichiCoder1 Any progress on using Buildx for this?

[RichiCoder1]

@DWSR I haven't put any active effort into this personal, though I think some work has happened via other contributers to make this happen. I haven't (and for the forseeable future) had time to work on this

[alinbalutoiu]

@DWSR I'm currently trying to build with docker buildx in GitHub Actions but the build is failing when building on arm64.

It looks similar to the issue described here, trying to debug the issue to see what's wrong.

[TrueBrain]

Been toying a bit with this. Just leaving this here as it might help someone else.

• install-packr-linux.sh needs some patching to work with Go 1.11+. I created a PR for this: fix: installing packr for arm is no longer working #5904.
• install-ksonnet-linux.sh no longer works, as there is a missing repository in the ksonnet dependency. See Remove ksonnet (solves #5256) #5271 for what to patch out to make the build work without ksonnet.

And I think the problem @alinbalutoiu ran into: when using qemu to emulate arm/arm64, you run into the issue that yarn cannot contact the Internet (neither can npm btw). It seems not all syscalls can be emulated, resulting in networking issues.
The solution seems to be to prepare a yarn cache via the native system, and share that with the build. How I approached this:

Before building, run on native system:

docker run --rm -it -v $(pwd)/ui/package.json:/src/package.json:ro -v $(pwd)/ui/yarn.lock:/src/yarn.lock:ro -v $(pwd)/cache:/cache --workdir /src node:12.18.4 yarn install --cache-folder /cache

I haven't found yet how to only download the files, so it will install everything. Shouldn't take long (17 seconds on my machine).

Now we have a cache, we can give this to the build step and build arm with this diff in Dockerfile:

+ADD cache /cache

-RUN yarn install
+RUN yarn install --offline --cache-folder /cache

The Docker context becomes huge, so not really ideal, but it does do the trick. Now building via docker buildx seems to work fine:

docker buildx build --platform linux/arm64 -t truebrain/argocd:v2.0.0-rc1 --build-arg BUILD_ALL_CLIS="false" .

This produces images; I have not tried them yet, that is next on the agenda :D

For anyone also trying this, also a few observations worth sharing:

• Building for arm/v7 fails for me on go get with TLS errors for any Google domain (works fine for github.com if using GOPROXY=direct). The reason seems to be that emulation is so slow, go get gives up before the TCP connection is established. It also takes ~60 CPU minutes to get to that point (where arm64 only takes a few CPU seconds to get there). This can fully be a problem of my Qemu version etc (using Ubuntu 20.04 via WSL2).
• The yarn steps take a while on arm64. This seems to be because it needs to compile a lot of these dependencies, as many don't seem to have an arm64 precompiled variant.
• webpack on native machines takes 133 seconds (enable timestamps). Via emulation, it takes even longer (800+ seconds for me). Be patient. It will get there in the end.

[mickkael]

Been toying a bit with this. Just leaving this here as it might help someone else.

• install-packr-linux.sh needs some patching to work with Go 1.11+. I created a PR for this: fix: installing packr for arm is no longer working #5904.
• install-ksonnet-linux.sh no longer works, as there is a missing repository in the ksonnet dependency. See Remove ksonnet (solves #5256) #5271 for what to patch out to make the build work without ksonnet.

And I think the problem @alinbalutoiu ran into: when using qemu to emulate arm/arm64, you run into the issue that yarn cannot contact the Internet (neither can npm btw). It seems not all syscalls can be emulated, resulting in networking issues.
The solution seems to be to prepare a yarn cache via the native system, and share that with the build. How I approached this:

Before building, run on native system:

docker run --rm -it -v $(pwd)/ui/package.json:/src/package.json:ro -v $(pwd)/ui/yarn.lock:/src/yarn.lock:ro -v $(pwd)/cache:/cache --workdir /src node:12.18.4 yarn install --cache-folder /cache

I haven't found yet how to only download the files, so it will install everything. Shouldn't take long (17 seconds on my machine).

Now we have a cache, we can give this to the build step and build arm with this diff in Dockerfile:

+ADD cache /cache

-RUN yarn install
+RUN yarn install --offline --cache-folder /cache

The Docker context becomes huge, so not really ideal, but it does do the trick. Now building via docker buildx seems to work fine:

docker buildx build --platform linux/arm64 -t truebrain/argocd:v2.0.0-rc1 --build-arg BUILD_ALL_CLIS="false" .

This produces images; I have not tried them yet, that is next on the agenda :D

For anyone also trying this, also a few observations worth sharing:

• Building for arm/v7 fails for me on go get with TLS errors for any Google domain (works fine for github.com if using GOPROXY=direct). The reason seems to be that emulation is so slow, go get gives up before the TCP connection is established. It also takes ~60 CPU minutes to get to that point (where arm64 only takes a few CPU seconds to get there). This can fully be a problem of my Qemu version etc (using Ubuntu 20.04 via WSL2).
• The yarn steps take a while on arm64. This seems to be because it needs to compile a lot of these dependencies, as many don't seem to have an arm64 precompiled variant.
• webpack on native machines takes 133 seconds (enable timestamps). Via emulation, it takes even longer (800+ seconds for me). Be patient. It will get there in the end.

instead of building the cache externally, I've used the following arg with yarn --network-timeout 100000
it works for me on a rpi4 host.

[TrueBrain]

instead of building the cache externally, I've used the following arg with yarn --network-timeout 100000
it works for me on a rpi4 host.

Nice, that works fine too indeed :) Much easier solution ;)

Small status update:

• Despite the fact fix: installing packr for arm is no longer working #5904 is merged, it did not make it into the v2 series (no clue why tbh :P). So don't be fooled by the "merged" status, for v2.0.3 you still need to apply that patch.
• Remove ksonnet (solves #5256) #5271 hasn't seen any new activity, so that too you still have to apply manually.

Other than that, this seems to work fine on a raspberry pi 4 cluster :)

[agustinvinao]

Hi,

I was trying to find any info about when arm64 images may be available. Any update on this or an idea when this may be available?

[Im5tu]

I'm also keenly interested in this. Anything that I can do to help push this along? :)

[johnlonganecker]

My Solution - Build with docker on Pi4 8G running Raspian

I realize this doesn't answer the issue, but when I google for arm images I get this page so I am including these directions to help future me or someone else on how to build and use an arm image

Install docker

curl -sSL get.docker.com | sh && \
  sudo usermod pi -aG docker

Built the image (this took awhile)

git clone https://github.com/argoproj/argo-cd.git
cd argo-cd
git checkout v2.1.2
make armimage
docker tag argocd:stable-arm johnlonganecker/argocd:v2.1.2
docker push johnlonganecker/argocd:v2.1.2

Installed with getting started yaml

wget https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Replaced container images with arm images in install.yaml

image: ghcr.io/dexidp/dex:v2.27.0      -> image: dexidp/dex:v2.27.0
image: quay.io/argoproj/argocd:v2.1.2  -> image: johnlonganecker/argocd:v2.1.2

kubectl apply -n argocd -f <modified-install-yaml>

Then it just worked!

[alborotogarcia]

image: ghcr.io/dexidp/dex:v2.27.0 -> image: dexidp/dex:v2.27.0

@johnlonganecker Looking at the github dex repo it seems that there are arm64 builds at v2.27.0.. what makes them different from the ones at docker registry?

I got it working on my pi4, though I am not sure if these are built on top of qemu, if so, then there's a big trade off in performance for sure..

[johnlonganecker]

@alborotogarcia When I left the image as ghcr.io/dexidp/dex:v2.27.0 it was not pulling an arm image and would fail to get into a running state. When I pulled it from docker hub IE dexidp/dex:v2.27.0 it just worked no problem. This is referencing the dexidp dockerhub account, I am not sure how they built their images (I am assuming that is what you meant by your last comment?).

I did not try throwing on the @ sha256 to get it to work, but that is worth trying! https://github.com/dexidp/dex/pkgs/container/dex/529012?tag=v2.27.0

[rdelpret]

Built a v2.1.3 Image yesterday on my 4 GB Raspberry pi and put it up on docker hub: rdelprete/argocd-arm64:v2.1.3

It took a very long time, I had to add --network-timeout 1000000 to all the yarn commands 😮

[robinhuiser]

I had to update the yarn install command with:

RUN for t in 1 2 3 4 5; do yarn install --network-timeout 1000000 && break || sleep 1; done

...since it would just not finish the first time (I am building argocd using Concourse pipelines deployed on a Microk8s Raspberry pi cluster).

Problem was... first time.. you do not have anything cached - hence (as many people are hitting the same issue) - you need to re-run "manually" the yarn install command a couple of times. Which is hard in a pipeline.