CI: Build/publish multi-architecture Docker images via ko

[aknuds1]

What this PR does:
Build and publish Docker images via ko, a tool originally made by Google for containerizing Go apps. I've forked ko to allow for storing images as an intermediate step, and then publishing those stored images in another step. The images get stored as an OCI image layout, since it's the most practical format.

The most important change, functionally speaking, is that multi-architecture Docker manifest lists get published (containing AMD64 and ARM64 images) for Mimir and tools (mimirtool, metaconvert, query-tee).

Removal of Dockerfiles

Since ko doesn't use Dockerfiles, I'm removing the Dockerfiles except for the build image one (still has to be built with Docker). I've also updated the Makefile to reflect this, hope I got everything right in this regard.

Testing of ko-based Docker manifest list

I've tested the AMD64 image produced by ko (ko pushes a multi-architecture manifest list, the Docker installation on my machine picks AMD64) by following the getting started guide, and it works well. The integration test suite also works.

Which issue(s) this PR fixes:

Checklist

• [na] Tests updated
• [na] Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[pstibrany]

Great work!

docker run us.gcr.io/kubernetes-dev/mimir-ko-86216e9e58680a21dc5c3f08aab8c895:chore-build-multiplatform-images-291766c22 --version
Mimir, version 1.10.0-rc.0 (branch: chore/build-multiplatform-images, revision: 291766c22)
  build user:
  build date:
  go version:       go1.17.5
  platform:         linux/arm64

🎉

I have few questions:

• What does our fork of ko add that's missing in upstream? Would it be possible to get our changes upstreamed?
• Can we make ko part of build image?
• Image publishing KO_DOCKER_REPO=us.gcr.io/kubernetes-dev ko publish-layout --tags ${IMAGE_TAG} mimir-ko mimir-images goes to us.gcr.io/kubernetes-dev/mimir-ko-86216e9e58680a21dc5c3f08aab8c895 repository. I don't understand where 86216e9e58680a21dc5c3f08aab8c895 part comes from, or how to change it.

[aknuds1]

Great work!

Thanks @pstibrany!

• What does our fork of ko add that's missing in upstream? Would it be possible to get our changes upstreamed?

The fork adds the ability to separate building and publishing into two different steps. I.e., the fork adds a sub-command publish-layout, that can publish an OCI image layout previously made by the build sub-command. publish-layout can publish to either an image registry or to a Docker daemon.

If we start using ko and it works well for us, I will ask upstream if it's interesting to them.

• Can we make ko part of build image?

Absolutely.

• Image publishing KO_DOCKER_REPO=us.gcr.io/kubernetes-dev ko publish-layout --tags ${IMAGE_TAG} mimir-ko mimir-images goes to us.gcr.io/kubernetes-dev/mimir-ko-86216e9e58680a21dc5c3f08aab8c895 repository. I don't understand where 86216e9e58680a21dc5c3f08aab8c895 part comes from, or how to change it.

This is because the -B (--bare) flag isn't used, without it ko appends an MD5 suffix to the repository name.

[pstibrany]

Pushed updated build-image with ko: us.gcr.io/kubernetes-dev/mimir-build-image:chore-build-multiplatform-images-bdfd88900.

[simonswine]

This is some great work 👍

I think the main gap from my point of view is that (according to my understanding) ko won't make any use at all of cmd/mimir/Dockerfile. Which might lead to some change between images before and after.

So I think lines like

EXPOSE     8080

and

RUN        apk add --no-cache ca-certificates

would need to be implemeted through ko (and I think you have done the ca-certificates using grafana/alpine:3.15.0).

I think ideally we should make use of ko images within GL as well, so we can get to the bottom of potential issues.

[aknuds1]

@simonswine I guess we can migrate away from the Dockerfile once we've eventually migrated to ko, yeah. I have implemented adding CA certificates via the base image yes. Unfortunately I found that ko's default base image breaks the integration tests due to lack of permission to read TLS certificates in the filesystem.

[aknuds1]

@simonswine do we need to replicate this from Dockerfile? Is it useful at all?

EXPOSE     8080

[simonswine]

@simonswine do we need to replicate this from Dockerfile? Is it useful at all?

EXPOSE 8080

I guess the only thing that uses it which I am aware of is docker run -P. So it might not be that impactful, I also think it could be added by adding it to the defaultBaseImage as ko doesn't appear to support it: ko-build/ko#472

[pstibrany]

Thank you for your work. After #1772 we will now publish multiarch images using Docker buildx and skopeo.

[aknuds1]

Thank you for your work. After #1772 we will now publish multiarch images using Docker buildx and skopeo.

OK, closing the PR.