arm64 support #90

alekitto · 2020-10-06T10:14:10Z

Related to #76

I needed arm based images for one of my work, so I modified the build scripts to build and push multiarch images to docker hub.
I'm publishing my work as a draft hoping that could be useful to support arm64 architecture.

The trickiest part was to test the build on my local environment, so I wrote the qemu_build script that installs qemu requirements, downloads the ubuntu image for the target platform and executes a build inside an emulated environment. It than exports the images into a tar, transfers it from the vm to the local environment and imports it as docker image with the right tag.
The script is not intended to run on CI environment like travis, as without KVM the build is very slow, but could be useful to test multi arch builds in local environments, or test new archs other than amd64 and arm64.

Hope this could help.

manelpb · 2020-10-18T16:00:19Z

please merge this, would really appreciate

kitos9112 · 2020-12-10T19:54:40Z

Any chance this could be reviewed? looks quite promising!

odidev · 2020-12-30T09:32:00Z

I have tried building docker images for arm64 with the changes suggested in this PR and able to build and use the arm64 docker image.

It would be helpful, if this PR is merged.

ddelange · 2021-01-09T04:10:19Z

We have decided on going forward with this and provide support for arm64 images. We still don't have an ETA for it, though, so thank you for your patience.

Originally posted by @alemorcuq in https://github.com/bitnami/bitnami-docker-redis/issues/192#issuecomment-752933612

Is there anything still needed here before Travis can start doing its thing?

james-w

Hi, overall this is very nice, thanks. I have some questions and comments, and one thing that definitely needs to be addressed before this can be used.

james-w · 2021-01-09T12:20:04Z

.travis.yml

+  after_success:
+    - 'if [[ "$DIST" == "buster" ]] ; then sudo docker tag "bitnami/minideb:$DIST-$PLATFORM" "$BASENAME:latest-$PLATFORM" ; fi'
+    - 'if [[ "$TRAVIS_BRANCH" == "master" ]] ; then sudo bash pushone $DIST $PLATFORM ; fi'


question: This is being done here as it's on a different machine and so it can't be pushed in the deploy stage?

Yes, exactly. Since every image is built on a different machine and there is no way to pass a docker image as artifact to another stage, the only way is to push the built image on after_success after the build is finished.

Correction: tar files created by buildone can maybe passed as artifact and pushed only in deploy stage.
Is there any limit to artifact files on travis?

I do not know.

One thing is that we run tests on the images after importing the tarball to the base image, which requires starting a container from the image. If we want to test all arches then that would have to be done on each arch I believe, which makes this the better way to do that, even with the pushing.

The alternative would be pushing images without tags to some holding registry, then having an artifact that reported the id and then doing the final push in deploy when everything is OK.

I'm not sure what is better, but this seems acceptable to me.

james-w · 2021-01-09T12:21:27Z

.travis.yml

+  after_success:
+    - 'if [[ "$DIST" == "buster" ]] ; then sudo docker tag "bitnami/minideb:$DIST-$PLATFORM" "$BASENAME:latest-$PLATFORM" ; fi'
+    - 'if [[ "$TRAVIS_BRANCH" == "master" ]] ; then sudo bash pushone $DIST $PLATFORM ; fi'
+    - 'if [[ "$DIST" == "buster" ]] ; then sudo bash pushone latest $PLATFORM ; fi'


suggestion (blocking): I think this also needs a guard that it is master branch as otherwise this will push the latest tag from PRs.

You're right 👍

james-w · 2021-01-09T12:22:26Z

.travis.yml

+            sudo docker manifest create bitnami/minideb:$DIST bitnami/minideb:$DIST-amd64 bitnami/minideb:$DIST-arm64
+            sudo docker manifest push bitnami/minideb:$DIST


question: does this require the images to be on this machine?

No. The docker manifest commands talks directly to the registry.
This is way the images bitnami/minideb:$DIST-amd64 and bitnami/minideb:$DIST-arm64 needs to be pushed with their own tags before this is executed.

james-w · 2021-01-09T12:24:39Z

buildone

+    docker tag "$built_image_id" "$BASENAME:$TAG-$PLATFORM"
+    log "Tagged $built_image_id as $BASENAME:$TAG-$PLATFORM"


question: is there some special handling in registries that means that this is understood to be a multiarch image? If not it seems like this means we are no longer going to update the existing tags?

The existing tags will be updated by the docker manifest push command as a multiarch tag, but the source images of the tag needs to be pushed before the docker manifest command could be issued.

I see, thanks.

james-w · 2021-01-09T12:25:46Z

pushone

+source <(curl -sSL "$CIRCLE_CI_FUNCTIONS_URL")
+# Use '.RepoDigests 0' for getting Dockerhub repo digest as it was the first pushed
+DIST_REPO_DIGEST=$(docker image inspect --format '{{index .RepoDigests 0}}' "${BASENAME}:${DIST}-${PLATFORM}")
+update_minideb_derived "https://github.com/bitnami/minideb-runtimes" "$DIST-${PLATFORM}" "$DIST_REPO_DIGEST"


question: have you checked that this will have the right behaviour? My guess is that this won't do anything because the platform tags aren't recognised.

Not really. And probably it won't work as expected reading the code on test-infra.
Probably it needs an update on test-infra, but should be executed after the deploy stage (after the docker manifest push command succeeds).

I guess that this should be moved out of pushone in to the deploy stage then? Even if the different arches just build the tarball and the pushing is done at the end it seems like we would want this bit to be run after the manifest step, so that all arches are present and the tags are updated? That would seem to be the best behaviour unless those scripts are modified to understand multi-arch so that they can update each arch separately, but that doesn't seem like it would be useful currently.

Now executing this on deploy stage on travis

james-w · 2021-01-09T12:28:23Z

qemu_build

+while do_sudo fuser /var/{lib/{dpkg,apt/lists},cache/apt/archives}/lock >/dev/null 2>&1; do
+    sleep 1
+done


suggestion: This is done in the travis file, so also doing it here seems weird, suggest removing it unless there's a reason it needs to be repeated here.

qemu_build file is not intended to be executed on travis.
I often encountered a lock issue while working on this executing apt-get update, so I copied this while loop from travis file. Is not really required, so can be removed.

Ah, I figured this out later and that makes sense, thanks.

james-w · 2021-01-09T12:29:02Z

qemu_build

+do_sudo apt-get update
+do_sudo apt-get install -y qemu-kvm libvirt-bin qemu-utils genisoimage virtinst curl rsync qemu-system-x86 qemu-system-arm cloud-image-utils


question: other dependencies are installed from the Makefile, is it being done here as these aren't always needed?

Is not needed in every case, but only if executing a build with qemu, so they are required to create a build environment (but not on travis for example).
Makefile install requirements inside the build environment (also on travis).

That makes sense. I would suggest making this another Makefile target? I'm not a fan of apt-get running every time you use a script as it wastes time once the dependencies are installed.

👍 added .installed-qemu make target

james-w · 2021-01-09T12:29:34Z

qemu_build

+mkdir -p .kvm-images/{amd64,arm64}
+
+if [[ ! -f .kvm-images/amd64/bionic-server-cloudimg-amd64.img && "$PLATFORM" == "amd64" ]]; then
+  curl -SL https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-amd64.img > .kvm-images/amd64/bionic-server-cloudimg-amd64.img


question: do Debian offer equivalent images?

Maybe debian openstack images could be used (https://cdimage.debian.org/cdimage/openstack/current/) but I'm not sure about it.

I think this is fine given that this is mainly for local testing. It just struck me a bit odd to be using Ubuntu to build Debian.

I just replaced the images with Debian ones.

james-w · 2021-01-09T12:30:23Z

qemu_build

+fi
+
+
+debian_snapshot_id=${3:-}


thought: not having this with the other params, and using lowercase where they use caps seems out of place

james-w · 2021-01-09T12:32:47Z

qemu_build

+do_ssh "apt-get update && apt-get install -y apt-transport-https debootstrap debian-archive-keyring jq dpkg-dev rsync ca-certificates curl gnupg-agent software-properties-common && mkdir /build"
+do_ssh "curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -"
+do_ssh "add-apt-repository \"deb [arch=$PLATFORM] https://download.docker.com/linux/ubuntu \$(lsb_release -cs) stable\""


thought: this duplicates some things that are done elsewhere. I wonder if copying over the Makefile and using that or something similar would make it easier to maintain?

Probably, trying to use Makefile to install common requirements now.

ddelange · 2021-01-10T13:15:21Z

Not sure if useful (maybe not for minideb but for the downstream repos building on it?), but for the record here is a neat example of multi-arch building using docker-tools' build_images I just came across:

joshuamorris3/namecheap-ddns-update#6

It seems docker-tools is available on Travis by default, and since alpine has all these archs available as well, building the multi-arch docker images using docker buildx is seamless.

The resulting docker tags have digests for all the different architectures:

https://hub.docker.com/r/joshuamorris3/namecheap-ddns-update/tags

james-w · 2021-01-13T13:01:40Z

I think there's a couple of things to fix, but I think a maintainer could mark this as ready for review now, and at least we could find out what Travis thinks of building the images.

james-w · 2021-01-13T17:25:23Z

Hi,

This is cool, thanks for your contribution, I think it looks good now. Unfortunately there's nothing more I can do to help at the moment. Hopefully a maintainer sees this and can move the PR forward.

Thanks,

James

beltran-rubo · 2021-01-13T18:50:39Z

Thanks a lot for the contribution & revisions @alekitto @james-w! And I apologise for the delay on getting back to you. We plan to review this PR as soon as possible and integrate into travis. We will keep you in the loop.

dani8art

Hi @alekitto great work here! my apologies for this late response.

It seems most of the questions were resolved and well handled by @james-w (thanks James). Mines are more in the snapshot feature and how it is going to be handle and some other restrictions with our Travis version.

Please, could you take a look at them?

dani8art · 2021-01-22T15:38:47Z

.travis.yml

+    - <<: *build_job
+      arch: arm64-graviton2
+      env:
+        - DIST=stretch PLATFORM=arm64


Unfortunately, I think we need to use arm64since arm64-graviton2 is only supported in .com our project is built in .org

https://docs.travis-ci.com/user/multi-cpu-architectures/#multi-cpu-availaibility

Didn't notice that build was running on .org. Fixed

Anyway on .org now there's a warning stating that will be shut down in a few weeks. If the build will be migrated to .com maybe we should revert to graviton2 which should have better performance

There could be a problem with arm64 architecture on travis-ci.org: it could run only in unpriviledged LXD container.
To correctly run debootstrap a full vm is needed, which is only available in arm64-graviton2.
I also tried to execute a QEMU build into a full amd64 vm, but it is extremely slow.

Stating that .org is about to shut down, maybe a migration to .com could be planned?

The third option is to try to build the images on GitHub Actions.
It shouldn't be too difficult to make a test workflow. The question is: will the execution be too slow?

The question is: will the execution be too slow?

where travis runs stages sequentially on the same machine (multiple machines also doable), github runs jobs in parallel on multiple machines by default. this should solve any speed issues probably?

docker provides a qemu setup action for multiarch GH workflows.

Im I'm not mistaken, GH by default runs up to four jobs on four runners (each runner 2CPU/7GB) concurrently (might be four not sure anymore) which can be decreased as well. all you'd need to do is define a job with a step that runs the qemu_build script with args ${{matrix.distro}} ${{matrix.other_arg}}, then define your custom matrix on the top of the job and you should be all set for parallel builds :)

for example, this CI runs in parallel on new commits to master, and all commits to all PRs (and emulates Travis's auto-cancel previous runs feature) , plus CD runs once upon a GitHub Release or Prerelease. as alternative to split files, add a push step to the job that only runs on a release/prerelease event:

on: pull_request: push: branches: master release: types: [released, prereleased] workflow_dispatch: # allows triggering manually from the Actions tab ... - name: Push image if: github.event_name == 'release' run: docker push

For the records: I just tried to build the images on GH Actions on my fork (master branch to test the push job), without using qemu_build as performance was too bad.
This is the last working run.

14 mins for buster arm64..

+1 for the effort though! plz dont delete the branch :)

fyi I bumped the build job to ubuntu-20.04 in the hope that this would run on newer github runners than the 18.04 job.

the result is about twice as fast (similar stages timings as compared to graviton2 555918d)

.travis.yml

README.md

juamedgod · 2021-02-09T13:48:58Z

.travis.yml

+      while sudo fuser /var/{lib/{dpkg,apt/lists},cache/apt/archives}/lock >/dev/null 2>&1; do
+        sleep 1
+      done
+    - sudo rm -f /usr/local/bin/jq


Why is this needed? (deleting jq)

The .installed-requirements make target fails without removing the travis-included jq executable

Uhm, which error was it throwing? If the package was already installed it should simply ignore it.

Don't remember the exact error message, but the jq executable is not installed in travis machine via apt.
Apt simply refuses to overwrite an existing file.

.travis.yml

juamedgod · 2021-02-09T15:34:58Z

.travis.yml

+      while sudo fuser /var/{lib/{dpkg,apt/lists},cache/apt/archives}/lock >/dev/null 2>&1; do
+        sleep 1
+      done
+    - sudo rm -f /usr/local/bin/jq


Uhm, which error was it throwing? If the package was already installed it should simply ignore it.

.travis.yml

juamedgod · 2021-02-10T18:57:25Z

.travis.yml

+
+            if [[ "$DISTS_WITH_SNAPSHOT" =~ (^|[[:space:]])"$DIST"($|[[:space:]]) ]] ; then
+              SNAPSHOT_NAME="$DIST-snapshot-$(./snapshot_id)"
+              sudo docker manifest create $SNAPSHOT_NAME:$DIST $SNAPSHOT_NAME:$DIST-amd64 $SNAPSHOT_NAME:$DIST-arm64


I think you meant $BASENAME:$SNAPSHOT_NAME (and the -amd64, -arm64 variants)

juamedgod · 2021-02-10T19:09:37Z

buildone

@@ -43,8 +43,9 @@ log() {

 build() {
    DIST=$1
+    PLATFORM=${2:-amd64}


I believe we should consider the $PLATFORM name when querying the registry in:

if docker pull "$BASENAME:$TAG" > /dev/null; then ... }

If not, when comparing the amd64 image (at least now that we only have a amd64 one, but is not labeled -amd64), the check at then end is probably going to determine the image is up to date:

else log "Image didn't change" return fi

And so it will skip tagging it:

docker tag "$built_image_id" "$BASENAME:$TAG-$PLATFORM"

This may not be a problem when all is up and running, but at least for the first time, it will fail because the tag does not exists when creating the manifest.

Maybe just appending it at the beginning would work:

if [ -n "$debian_snapshot_id" ]; then TAG="${DIST}-snapshot-${debian_snapshot_id}-$PLATFORM" else TAG=$DIST-$PLATFORM fi

That would also require tweaking the "test" script to check the image name correspond to jessie. From:

if [ "jessie" == "$DIST" ]; then

to something like

if [[ "$DIST" == "jessie"* ]]; then

juamedgod · 2021-02-10T19:11:51Z

.travis.yml

+    - stage: deploy
+      if: branch = master AND type = push
+      env:
+        - DISTS="stretch buster latest"


As jessie does not get a multiplatform manifest and we are tagging it as jessie-amd64, we should add some extra logic to tag it as simply jessie too, for backwards compatibility.

Right, I've added a build job for jessie image push.
Probably there's something to be corrected in update_minideb_derived to check multiple digests, but is out of scope of this PR.

juamedgod · 2021-02-16T12:53:06Z

@alekitto Ok, I believe this is ready to land :). Thanks a lot for your help adding this feature and for your responsive answers to our feedback!

alekitto · 2021-02-16T13:39:01Z

Sorry, there's an error in buildone script (lines 57, 99 and 100). It should be $TAG instead of $TAG-$PLATFORM. Should I open a new PR?

juamedgod · 2021-02-16T14:33:13Z

I already fixed 99 and 100 but missed 57. Don't worry, we are handling it. Thanks for offering!

alekitto force-pushed the multiarch_build branch from 6d6ca05 to 2996fd9 Compare October 6, 2020 10:50

ddelange mentioned this pull request Jan 8, 2021

Support ARM/multi-arch images for Kubeapps containers vmware-tanzu/kubeapps#929

Closed

james-w suggested changes Jan 9, 2021

View reviewed changes

alekitto force-pushed the multiarch_build branch 2 times, most recently from a8f347a to 7779615 Compare January 13, 2021 12:23

alekitto force-pushed the multiarch_build branch from 7779615 to 1745656 Compare January 13, 2021 14:13

arm64 support

deb0332

alekitto force-pushed the multiarch_build branch from 1745656 to deb0332 Compare January 13, 2021 17:15

alekitto marked this pull request as ready for review January 13, 2021 17:28

dani8art requested changes Jan 22, 2021

View reviewed changes

alekitto added 3 commits January 22, 2021 18:36

build buster snapshot images and fix arch for travis-ci.org

1b01af7

add paragraph for qemu_build on README

980da68

fix build on arm

15bc96a

alekitto force-pushed the multiarch_build branch 7 times, most recently from 2ce5677 to 1ad9efe Compare January 25, 2021 16:06

temporarily restore arm64-graviton2 as build arch

555918d

alekitto force-pushed the multiarch_build branch from 1ad9efe to 555918d Compare January 26, 2021 16:37

juamedgod reviewed Feb 9, 2021

View reviewed changes

alekitto added 2 commits February 9, 2021 15:48

fix build script in travis.yml

5a6e227

use global env BASENAME for docker image name

e15de67

juamedgod reviewed Feb 9, 2021

View reviewed changes

use LATEST and DISTS_WITH_SNAPSHOT env vars in travis configuration

aeabf3b

juamedgod reviewed Feb 10, 2021

View reviewed changes

fix jessie builds and repo digest fetch command

3e8a707

juamedgod merged commit e44e9f1 into bitnami:master Feb 16, 2021

manelpb mentioned this pull request Mar 7, 2021

Possible to get an ARM image? vmware-archive/kubewatch#208

Open

alemorcuq mentioned this pull request Mar 8, 2021

MariaDB ARM64 version bitnami/charts#4825

Closed

carrodher mentioned this pull request Aug 24, 2021

Support for ARM64 architecture in Bitnami container images bitnami/charts#7305

Closed

migruiz4 mentioned this pull request Nov 5, 2021

Arm32 or Arm/v7 support #114

Closed

		sudo docker manifest create bitnami/minideb:$DIST bitnami/minideb:$DIST-amd64 bitnami/minideb:$DIST-arm64
		sudo docker manifest push bitnami/minideb:$DIST

		docker tag "$built_image_id" "$BASENAME:$TAG-$PLATFORM"
		log "Tagged $built_image_id as $BASENAME:$TAG-$PLATFORM"

		do_sudo apt-get update
		do_sudo apt-get install -y qemu-kvm libvirt-bin qemu-utils genisoimage virtinst curl rsync qemu-system-x86 qemu-system-arm cloud-image-utils

		fi


		debian_snapshot_id=${3:-}

arm64 support #90

arm64 support #90

Conversation

alekitto commented Oct 6, 2020

manelpb commented Oct 18, 2020

kitos9112 commented Dec 10, 2020

odidev commented Dec 30, 2020

ddelange commented Jan 9, 2021

james-w left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ddelange commented Jan 10, 2021 • edited Loading

james-w commented Jan 13, 2021

james-w commented Jan 13, 2021

beltran-rubo commented Jan 13, 2021

dani8art left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ddelange Jan 26, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ddelange Jan 28, 2021 • edited Loading

Choose a reason for hiding this comment

juamedgod Feb 9, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alekitto Feb 9, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

juamedgod commented Feb 16, 2021

alekitto commented Feb 16, 2021

juamedgod commented Feb 16, 2021

ddelange commented Jan 10, 2021 •

edited

Loading

ddelange Jan 26, 2021 •

edited

Loading

ddelange Jan 28, 2021 •

edited

Loading

juamedgod Feb 9, 2021 •

edited

Loading

alekitto Feb 9, 2021 •

edited

Loading