Add support for other architectures

[vdemeester]

Kubernetes support more architecture than x86_64, Tekton Pipeline should support those too.

What this means is :

• the project compiles on the architecture (and tests runs too)
• controller and webhook should be able to run on supported architecture
• images like entrypoint, creds-init, git-init, … should also be able to run on those architectures
• container images published from tekton (releases or nightly) should be multi-arch images (aka 1 reference for multiple images chose depending on the architecture)
• bonus : e2e test on cluster of these architectures

Architecture to support

• arm64 : an initial take on this has been done by @lisa, see here
• ppc64
• s390x
• and maybe of course windows 💃

Architecture supported

• arm64 : an initial take on this has been done by @lisa, see here
• ppc64
• s390x
• and maybe of course windows 💃

Marking this as an epic as it is a big chunk of work and is gonna need to be split up into several issues.

[lisa]

As part of my investigation that @vdemeester mentioned I had to change the base images in .ko.yaml*, even though busybox has an arm64 manifest. I believe this is because in google/ko's commands/config.go which exhibits underlying behaviour in go-containertools (described GoogleContainerTools/kaniko#646 as it relates to kaniko) to not pick image manifests according to what arch is currently making the request. In short, this issue likely blocks on either go-containerregistry doing something different with its default platform or having ko passing the desired platform along as in the kaniko PR.

[vdemeester]

cc @imjasonh @mattmoor (for the ko part)

[imjasonh]

ko-build/ko#6 cc @jonjohnsonjr

[jonjohnsonjr]

I added some more details to that bug with what I had in mind.

[vincent-pli]

@vdemeester @imjasonh
What about this project? do we have ppc64 supported?

[vdemeester]

@vincent-pli as of today, we only support linux/amd64. We should aim to support more combinaison of GOOS and GOARCH though.

[mattmoor]

The container tools nerd in me would love to see this happen, but I'm concerned about our current testing capability for this matrix.

@jonjohnsonjr can probably guide someone through this (he has an "ugly" PoC), but we probably won't be able to prioritize this before Knative Serving "v1"s.

[jonjohnsonjr]

Published what I was playing with here: ko-build/ko#38

If anyone's interested in picking this up, I'm happy to chat.

[vincent-pli]

@lisa
I saw your article: https://thedoh.dev/kubernetes/tektoncd-pipeline.html
Seems you did not change the defaultBaseImage in .ko.yaml, did the controller work?

[lisa]

@lisa
I saw your article: https://thedoh.dev/kubernetes/tektoncd-pipeline.html
Seems you did not change the defaultBaseImage in .ko.yaml, did the controller work?

@vincent-pli Yes, it worked. The build process (./hack/release.sh) built a controller image which I'm storing as thedoh/arm64-tektoncd-pipeline-cmd-controller.

[skaegi]

I know there is work in-house on doing a Tekton for s390x. Is the idea here that our build/release process would also have images for these other architectures (oops... nevermind I see one image to rule them all above)? What are we looking for here then? Do we want a tick mark when we can confirm Tekton releases works on a particular architecture?

[apasrini]

@skaegi We are right now trying Tekton for s390x. Is it supported or it is yet to be complete? Or is it available in a branch so that we could try?

[vdemeester]

@apasrini It's not "officially" supported, as "we do not ship s390x images" but it should be possible. We (Red Hat & IBM) are working on having OpenShift Pipelines supporting s390x. In parallel we should work upstream to make sure it supports it to (cc @barthy1). To be able to build and release s390x (and other platforms) for tekton, we depend on ko-build/ko#38 — or we should modify our release process to be able to not use ko for other architectures in the meantime.

@afrittoli @imjasonh @sbwsg @dibyom we also could try to use a patch version of ko (based on ko-build/ko#38) to try this out (and gather feedback for ko so that it gets in at some point).

[barthy1]

@vdemeester I've played a little bit with the latest ko version (without taking into account ko-build/ko#38)
A little bit strange behaviour takes place, but there is some dirty way to use ko for building s390x even now. So what I can see:

1. s390x host and s390x base image in .ko.yaml (defaultBaseImage: docker.io/s390x/busybox:latest) -> s390x image and "Architecture": "s390x" in manifest
2. s390x host and multi-arched base image in .ko.yaml (defaultBaseImage: alpine:latest) -> s390x image and "Architecture": "amd64" in manifest
3. amd64 host and s390x base image in .ko.yaml (defaultBaseImage: docker.io/s390x/busybox:latest) -> amd64 image and "Architecture": "s390x" in manifest
4. amd64 host and multi-arched base image in .ko.yaml (defaultBaseImage: alpine:latest) -> amd64 image and "Architecture": "amd64" in manifest

Result: First option provides a way to have s390x image build with current ko, but it requires to have s390x hardware

[mattmoor]

I'd love to get Jon's PR moving forwards, and we were just talking last week about some thoughts here. Let's discuss there?

[easmithdev]

Is there any update surrounding ARM64 support for the tekton controller and webhook images, as well as pipeline workloads? ARM64 should be 1st class citizen in regard to architecture support.

[jonjohnsonjr]

ko-build/ko#38 has landed which should unblock this, if anyone wants to jump on it.

Also happy to hear any feedback if that's not sufficient. I still need to write docs and stuff but it's pretty straightforward to play around with:

1. Set a multi-platform image as the base image.
2. Use --platform=all

[docularxu]

I did a native Arm64 build of teckton/pipeline on Arm64 servers. Also, I had docker images built, everything is run and tested on Arm64. Is there anyway I can contribute my work to you?

The description of my work can be found here: Build tekton-v0.15.2 on Arm64.md

For the tests, I run a lighthouse+tekton pipeline.

[julweber]

+1, see #4202 also

[ankuo]

Is there any update surrounding ARM64 support for the tekton controller and webhook images, as well as pipeline workloads? Whether a related image is provided

[mattmoor]

arm64 should work just fine. I generally build my own distro downstream, but I have been running on RPi4 and Graviton clusters without any known issues with Tekton. I'd open issues for anything you see.

[mattmoor]

(that said, I'm pretty sure upstream ships at least amd64/arm64, but likely also s390x/ppc64le/arm)

[ankuo]

@mattmoor Is there an official mirror for arm64 version available

[mattmoor]

All of the images should be multi-arch, from latest release:

# crane manifest gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/controller:v0.33.1@sha256:590777b63ef2ab6027cb819c5f81f9bd32f1e38ca2e8a5da8115e0603db96f85 | jq .
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
  "manifests": [
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "size": 756,
      "digest": "sha256:56d4c294506151f6447eedd18ece7d75942e5a7ef0a37bfba0bd2d3f31d50a7b",
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "size": 756,
      "digest": "sha256:e9eb0e795b17070e3b8d9290b3c0543a98cf510f6b4cdc723eca79e7ed2f4877",
      "platform": {
        "architecture": "arm",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "size": 756,
      "digest": "sha256:cc6e03bd33208231c241345d7b6c4c681027f53fd0cb0a604d232e8f23d114e1",
      "platform": {
        "architecture": "arm64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "size": 756,
      "digest": "sha256:f061293cdc773c8cd41e5e031e58f7fde2269244626ac8bed864e7566b8005a1",
      "platform": {
        "architecture": "ppc64le",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "size": 756,
      "digest": "sha256:c625d63d1893128593d0ed1e024ffee3ae5f964a1d69a4501e4988ad272e9686",
      "platform": {
        "architecture": "s390x",
        "os": "linux"
      }
    }
  ]
}

[ankuo]

@mattmoor Thanks for getting back to me so quickly. Forgive me. I don't know how to get the download link corresponding to these images, and how to find the download link corresponding to the historical version

[mattmoor]

The images referenced from release.yaml here are all multi-architecture images, which means your cluster will download the appropriate images based on its architecture.

In short: the amd64 yaml is the arm64 yaml since the images all support both architectures.

Hopefully that clarifies?

[ankuo]

@mattmoor This is so cool！ I gave it a try and it worked. Thank you again. Thank you.

[afrittoli]

We do have nightly tests for ppc64 and s390x as we have plans to add PR testing as well.
We currently have no infrastructure available to test amd64 and windows.

[vdemeester]

We do have nightly tests for ppc64 and s390x as we have plans to add PR testing as well. We currently have no infrastructure available to test amd64 and windows.

@afrittoli You mean aarch64 or arm64 (amd64 being our "default") 😝

[imjasonh]

We currently have no infrastructure available to test amd64 and windows.

Both linux/arm64 and windows/amd64 are available on GKE, if an enterprising someone felt like making those nodes available in a testing or dogfood cluster. 😄

[jerop]

@vdemeester now that we've added support for windows, can we consider this issue completed? (triaging old issues)

[vdemeester]

Probably, although " bonus : e2e test on cluster of these architectures" is not fixed, meaning, we know it compiles for all those architectures, but we do not really know if they actually works (by runinng e2e tests on them).