ARM64 (AARCH64) Support (Both Docker and Linux binaries)

[JasonSwindle]

Summary

Support for ARM64 would be grand.

[jakubdyszkiewicz]

Hi Jason,
I think we can talk about it once Envoy has official ARM64 images. I see there is some work happening at the moment envoyproxy/envoy#1861
We're happy to accept contributions. What is your use case?

[JasonSwindle]

Howdy,

Many different cloud providers now have ARM64 instances, and finding a service mesh that can work in a home lab setup is really dang hard. Devices like the RockPro64 / Rock Pi 4B are awesome SBCs to run containers on. I hope this helps. :)

[joakimr-axis]

Second that, and I would suggest also armv7. I see the case where you want to do hybrid cloud with both insances from the big providers and your own various (meaning not only x86_64) devices on-prem. IMO that is a prerequisite for claiming the universal part in the "Universal Service Mesh" tagline. (Now, certainly Kuma is in an early stage so please consider this community feedback for future steps rather than rag-down critic.)

[subnetmarco]

@jakubdyszkiewicz @nickolaev is this something that we can implement at this time? I believe that Envoy does ship with ARM64 images now.

[nickolaev]

From Kuma point fo view, compiling binaries for AArch64 is trivial, however we have 2 challenges:

• I don't find pre-compiled aarch64 binaries, we have to compile these on our own
• CI testing. CircleCI does not provide runners for AArch64. We can use qemu for cross-compilation and some basic tests, but that is very very slow

So overall yes, it needs some efforts though. Anyone from the community willing to take on this one?

[billyshambrook]

CircleCI supports ARM now - https://circleci.com/docs/2.0/arm-resources/

[github-actions]

This issue was inactive for 30 days it will be reviewed in the next triage meeting and might be closed.
If you think this issue is still relevant please comment on it promptly or attend the next triage meeting.

[lahabana]

On darwin building envoy:

From @lobkovilya:

I tried to build envoy using this unusual command

brew install --build-from-source envoy --HEAD

I didn’t check if it support git tags, also I didn’t try to build using bazel. But in the end, I think it should be quite simple, maybe half a day at max.

It would be nicer to use bazel for sure to make it simpler

[lahabana]

Full work:

1. See which tools don't work on arm64? Is it only on Mac M1?
2. Build images
3. Get envoy builds? Envoy builds for arm64 #4158
4. Run e2e tests on arm64
5. Publish images

[lukidzi]

I've tried to run make build, make dev/tools/all, make test and make test/e2e E2E_PKG_LIST=<few different tests>.
Looks like some things are working but not everything. We might have to contribute or check and replace current tools with something similar. Below I've posted problems that I had.

Darwin M1 ARM

make dev/tools/all

# missing `protoc-3.14.0` build for ARM64 Darwin ✅
- we can upgrade to 3.20.0(check changes) - since this version there is a build

# missing Kubectl 1.18.14 build for ARM64 Darwin ✅
- we can upgrade to 1.21+ there are builds with ARM64 Darwin 

# missing Kubebuilder build for ARM64 Darwin
-  on Darwin I've downloaded the package from Makefile for AMD and tried to run binaries, they worked except `etcd` which requires ARM64 build

# missing shellcheck build for ARM64 Darwin
- there is no published package but it's possible to install by brew - brew install shellcheck

make build

# missing Envoy binary
- we need to publish it

make e2e/test

# Helm charts have hardcoded node selector: `linux/amd64` <- for some tests fails
- we can parametrize
# `bitnami/kubectl:1.20` image doesn't work on ARM64 https://github.com/bitnami/bitnami-docker-kubectl/issues/22

Linux arm64

make dev/tools/all

# Kubebuilder for Linux ARM doesn't come with `kube-apiserver/etcd`
- we can make another step that installs `etcd/kube-apiserver`

make e2e/test

# Helm charts have hardcoded node selector: `linux/amd64` <- for some tests fails
- we can parametrize
# `bitnami/kubectl:1.20` image doesn't work on ARM64 https://github.com/bitnami/bitnami-docker-kubectl/issues/22

Kubebuilder package

Kubebuilder package which is downloaded during make dev/tools/all comes with:

• etcd - 3.3.11
• kube-apiserver - v1.16.4

[lahabana]

For bitnami/kubectl we can potentially just use rancher's instead: https://hub.docker.com/r/rancher/kubectl/tags

[lahabana]

For etcd we may want to install it separately: https://github.com/etcd-io/etcd/releases/tag/v3.5.2