Docker image: Multi arch support #7397

Starbix · 2020-05-03T13:51:00Z

Pull Request Checklist

Pull request is based on the develop branch
Pull request includes a changelog file. The entry should:
- Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
- Use markdown where necessary, mostly for code blocks.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
Pull request includes a sign off
Code style is correct (run the linters)

This PR updates the CircleCI configuration to use docker buildx to build the docker image. This way multi-arch images can be built on amd64. Currently only amd64 and arm64v8 are enabled. But other architectures such as arm/v7 or even s390x can be easily added.

Building for amd64 and arm64v8 took around 1h 30m on a Docker Medium instance on CircleCI.

Closes #4599

In case you want to test it, the multiarch image created by CircleCI is available under starbix/synapse:latest (master branch).

Signed-off-by: Cédric Laubacher [email protected]

babolivier

Thanks for your contribution!

It looks like both jobs are almost doing the same thing, with a consequent amount of instructions which would make this config harder to maintain in the long run.

I'd suggest moving that to a bash script (e.g. in .circleci) so that each job can just have one run instruction which could look like .circleci/dockerhub_upload [TAG] or something like that.

Starbix · 2020-05-09T13:49:50Z

I've added CircleCI commands to remove duplicate code

babolivier · 2020-05-11T08:58:47Z

This looks globally sane to me, however when I try to run it in CircleCI it seems to be stuck on the pip installs. I'll try to find some time to investigate why.

babolivier · 2020-05-11T09:00:55Z

Also you can fix the CI by pulling and merging the develop branch into your branch.

Starbix · 2020-05-11T09:02:05Z

This looks globally sane to me, however when I try to run it in CircleCI it seems to be stuck on the pip installs. I'll try to find some time to investigate why.

How long did you let it run for? It may just be slow

babolivier · 2020-05-11T09:04:10Z

It's currently been running for 14min.

babolivier · 2020-05-11T09:05:23Z

Oh wait it is indeed very slow, the amd64 passed that step, now it's just waiting for the arm64 one. I'll let it run and see what happens.

…multi-arch

Starbix · 2020-05-11T09:13:14Z

It can take 1.5 hours. Maybe only enabling multi-arch for releases would make sense.

anoadragon453 · 2020-05-21T13:41:11Z

We've decided this is something we want, but are aware that it would delay docker images after a release is made. We need to see whether this can be sped up at all, but if not we'll probably still do it.

Starbix · 2020-05-26T11:58:58Z

Using a native arm64 machine as a self-hosted runner would speed things up.

richvdh

I've started taking a look at this; a few initial comments and questions below.

Incidentally, I found https://www.docker.com/blog/multi-arch-build-and-images-the-simple-way/ a helpful introduction here.

Anyway, the problem here is the build time. I think the problem is simply that building all the arm stuff in an emulated arm environment is very slow. (for example: Building wheel for cryptography (PEP 517) takes almost 10 minutes).

I don't think it's appropriate that it should take an hour and a half for amd64 images to be built and published after each release simply to make it easier to run the images on arm, so we can't land this as-is.

A few thoughts on what we can do instead:

One option would be to build and push the amd64 and arm images separately, so that the amd64 users aren't made to suffer for the sake of the arm users. I don't know how easy that is to do.
We could switch the docker images away from alpine, which would mean we could take advantage of prebuilt manylinux wheels rather than having to build a load of stuff ourselves. Update the dockerfile to be Debian based, not Alpine based #6373 started this, but confused the issue with unrelated changes and eventually got abandoned. A clean start on this would be welcome.
Using a native arm64 machine would certainly help, but setting one up is going to be a whole bunch of work; I'd prefer to keep that as a last resort.

richvdh · 2020-07-03T13:26:48Z

.circleci/config.yml

    steps:
      - checkout
-      - run: docker build -f docker/Dockerfile --label gitsha1=${CIRCLE_SHA1} -t matrixdotorg/synapse:${CIRCLE_TAG} -t matrixdotorg/synapse:${CIRCLE_TAG}-py3 .
+      - setup_remote_docker:
+          version: 18.09.3


any reason to pin the docker version here? Indeed my impression is that buildx works better with 19.03?

Sorry for not responding earlier, I wasn't near a computer for the last 2 weeks.
If I remember correctly, 18.09 is the highest version CircleCI supports and it defaulted to a lower version.
A higher version would be beneficial, as there wouldn't be a need to create context.

The newest version available seems to be 19.03 now: https://circleci.com/docs/2.0/building-docker-images/#docker-version
I'll be able to test it and make the other changes this week.

richvdh · 2020-07-03T14:04:25Z

.circleci/config.yml

+      - run: curl --silent -L "https://github.com/docker/buildx/releases/download/<< parameters.buildx_version >>/buildx-<< parameters.buildx_version >>.linux-amd64" > ~/.docker/cli-plugins/docker-buildx
+      - run: chmod a+x ~/.docker/cli-plugins/docker-buildx
+      - run: docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
+      - run: docker context create old-style


why is the context called old-style ? (indeed, why do we need to create a context here?)

The name could be anything, but it was often called old-style in the tutorials I read. Creating a context is needed as CircleCI isn't running a relatively modern version of Docker.

richvdh · 2020-07-03T14:06:19Z

.circleci/config.yml

+      dockerfile:
+        type: string
+        default: docker/Dockerfile
+      push:
+        type: boolean
+        default: true


is there any point in parameterising these? it seems to complicate the commandline without any real benefit.

No, not at this moment. I added these parameters so one could easily expand the pipeline. I can remove the unused parameters.

richvdh · 2020-07-03T14:11:41Z

.circleci/config.yml

+        type: string
+        default: --label gitsha1=${CIRCLE_SHA1}
+    steps:
+      - run: docker buildx build -f << parameters.dockerfile >><<# parameters.push >> --push<</ parameters.push >> --platform << parameters.platforms >> << parameters.parameters >> << parameters.context >>


the default output format does not work well in circleci. suggest --progress=plain.

richvdh · 2020-07-03T14:39:14Z

(link to test circleci build: https://app.circleci.com/pipelines/github/matrix-org/synapse/13909/workflows/b387ce2c-5230-4505-8bea-e8dad635adad/jobs/25549)

As mentioned in #7397, switching to a debian base should help with multi-arch work to save time on compiling. This is unashamedly based on #6373, but without the extra functionality. Switch python version back to generic 3.7 to always pull the latest. Essentially, keeping this as small as possible. The image is bigger though unfortunately.

richvdh · 2020-07-17T16:43:33Z

Now that #7839 has landed, I've kicked off another test build at https://app.circleci.com/pipelines/github/matrix-org/synapse/14292/workflows/0c35f3e6-05fe-42e3-a955-30712a8c13fa/jobs/25558. I'll be interested to see how long it takes this time.

richvdh · 2020-07-21T12:27:21Z

I'll be interested to see how long it takes this time.

answer: it failed, for missing header files. Looks like more changes are needed to support arm64 builds here.

richvdh · 2020-08-06T17:33:19Z

I think this is superceded by #7921, so I'm going to close it.

Add multi-arch support (amd64, arm64v8)

c195eaa

Starbix mentioned this pull request May 3, 2020

Add aarch64 docker image #4599

Closed

babolivier suggested changes May 7, 2020

View reviewed changes

Starbix force-pushed the multi-arch branch 3 times, most recently from 906824f to eee30ac Compare May 9, 2020 13:48

Clean up CI

0a55a79

Starbix force-pushed the multi-arch branch from eee30ac to 0a55a79 Compare May 9, 2020 13:56

Starbix added 3 commits May 11, 2020 11:10

Add multi-arch support (amd64, arm64v8)

1433c17

Clean up CI

c5047c8

Merge branch 'multi-arch' of https://github.com/starbix/synapse into …

07a64a7

…multi-arch

erikjohnston assigned babolivier Jun 11, 2020

richvdh self-requested a review July 2, 2020 16:13

richvdh suggested changes Jul 3, 2020

View reviewed changes

richvdh unassigned babolivier Jul 3, 2020

richvdh added the X-Awaiting-Changes A contributed PR which needs changes and re-review before it can be merged label Jul 3, 2020

maquis196 mentioned this pull request Jul 13, 2020

Switch to Debian:Slim from Alpine for the docker image #7839

Merged

4 tasks

richvdh closed this Aug 6, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docker image: Multi arch support #7397

Docker image: Multi arch support #7397

Starbix commented May 3, 2020 •

edited

Loading

babolivier left a comment

Starbix commented May 9, 2020

babolivier commented May 11, 2020

babolivier commented May 11, 2020

Starbix commented May 11, 2020

babolivier commented May 11, 2020

babolivier commented May 11, 2020

Starbix commented May 11, 2020

anoadragon453 commented May 21, 2020

Starbix commented May 26, 2020

richvdh left a comment

richvdh Jul 3, 2020

Starbix Jul 20, 2020

Starbix Jul 20, 2020

richvdh Jul 3, 2020

Starbix Jul 20, 2020

richvdh Jul 3, 2020

Starbix Jul 20, 2020

richvdh Jul 3, 2020

richvdh commented Jul 3, 2020

richvdh commented Jul 17, 2020

richvdh commented Jul 21, 2020

richvdh commented Aug 6, 2020

Docker image: Multi arch support #7397

Docker image: Multi arch support #7397

Conversation

Starbix commented May 3, 2020 • edited Loading

Pull Request Checklist

babolivier left a comment

Choose a reason for hiding this comment

Starbix commented May 9, 2020

babolivier commented May 11, 2020

babolivier commented May 11, 2020

Starbix commented May 11, 2020

babolivier commented May 11, 2020

babolivier commented May 11, 2020

Starbix commented May 11, 2020

anoadragon453 commented May 21, 2020

Starbix commented May 26, 2020

richvdh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

richvdh commented Jul 3, 2020

richvdh commented Jul 17, 2020

richvdh commented Jul 21, 2020

richvdh commented Aug 6, 2020

Starbix commented May 3, 2020 •

edited

Loading