Add initial support for ARM64

[hakman]

This adds support for ARM64 worker nodes. Masters still require AMD64.

Fixes #8932
Fixes #6140

[hakman]

/assign @justinsb @mikesplain @rifelpet

[hatemosphere]

M6g are in GA now, so this could bring even more value now...

[hakman]

@hatemosphere should be part of Kops 1.19. At the moment, I am waiting for the branch to be created to get started with the review and various improvements.

[mikesplain]

@hakman This looks pretty good, you definitely need to rebase though before we can do a final review. This is quite a big PR... from the looks, there's not much we can do to break it up but it would be great if you thought of ways to make this more easily reviewable.

[hakman]

Thanks @mikesplain. I've been also thinking of ways to break this up. I think things changed quite a bit since I worked on this PR. Still waiting for the 1.18 branch to be created to do a rebase and go ahead with the review.

[hakman]

Setting as WIP until #9216 is merged and this can be simplified. Code already changed too much for a simple rebase.

[hakman]

Had to to one more rebase to restart the failed Netlify tests.
Thanks @mikesplain!

[mikesplain]

/lgtm

[lubinszARM]

Hi @hakman
A quick question.
I'm from Arm.
Currently we also want to run prow jobs on an arm64 cluster.
In the following file, I can see that you have created similar jobs.
https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes/kops/kops-periodics-misc.yaml#L106

But I'm curious that, it seems gcr.io/k8s-testimages/kubekins-e2e:v20200623-2424179-master/bootstrap/sidecar/initupload/clonerefs does not support Arm64.
So, how did you run the test on an arm cluster?

Reference chart of our prow test for Arm64:

Related issue:
kubernetes/test-infra#16588
kubernetes/kubernetes#86295

[hakman]

@lubinszARM in this context, our setup just uses Kops to create a cluster in AWS and run tests remotely on it. Kops takes care of everything except running the test.

[lubinszARM]

/cc @zhlhahaha

[zhlhahaha]

Hi, @hakman
I just read the PR kubernetes/test-infra#18034
For the job "e2e-kops-aws-misc-arm64", it uses image "gcr.io/k8s-testimages/kubekins-e2e:v20200623-2424179-master". However, the image is not support arm64 platform yet.
But, according to the result of the prow jobs, it looks working well. https://k8s-testgrid.appspot.com/sig-cluster-lifecycle-kops#kops-aws-arm64
How it works?

[hakman]

Hi @zhlhahaha @lubinszARM.

That image is started on AMD64 setup and runs a Kops binary to create a remote cluster in AWS. When that cluster is ready the e2e tests are run on the remote cluster: https://github.com/kubernetes/kops/blob/master/docs/development/testing.md.
Kops does things a bit differently than normal tests, so it is very flexible to try multiple screnarios, (different CNI plugins, different runtimes, ...).

So, there is no need in our case for a multi-arch image for kubekins-e2e, just support in tests. As you can see, there are very few issues remaining there.

There are also some issues with the build marker that is updated before the cross-build is done and tests fail because the ARM64 CI binaries are missing. There is a discussion about this in #sig-testing and hopefully there will be a solution soon.

Thanks for the referenced issues. Just browsed them, but will make some interesting read later.

Nice to see that Arm is pushing things in this direction!

[zhlhahaha]

Hi @hakman, I have read through the testing.md and it is quit interesting.
Thanks for your explaination!

[atam003]

@hakman
Will these changes work for kops for 1.18. if I try to back merge in the workspace.
I don't want to move to kops 1.19 as I see a few breaking (not backward compatible) changes (also it is a beta version).

[hakman]

@atam003 these changes will not work in kOps 1.18, there are too many changes.
kOps 1.19 will be released today so, no longer beta. In case it helps, you can use k8s 1.18 with it and enable ARM64 also.

[atam003]

@hakman
I still see the kops 1.19 beta version in Github. Any idea when it will be updated on GitHub or where should I look.