Add support for buildkit, for building to containerd

[afbjorklund]

Currently we support either "docker" or "podman" for building, either locally on the VM or remotely through the "env" commands.

https://minikube.sigs.k8s.io/docs/handbook/pushing/

When using the containerd container runtime, we currently don't have any support for building images - only for loading them...

We could include buildkitd, in order to provide that ?

---

It is used in the same way as docker or podman (through ssh), but with the buildctl client.

Similar to when using containerd, the syntax is somewhat more low-level than the other tools:

buildctl build --frontend=dockerfile.v0 --local context=. --local dockerfile=.

https://github.com/moby/buildkit

[afbjorklund]

The "normal" way to invoke this would be docker buildx or DOCKER_BUILDKIT=1 docker build.

The above goes for when not having a dockerd running, but only using stand-alone containerd.

The idea would be to use the containerd worker for buildkitd, and to use the containerd output...

sudo buildkitd --containerd-worker true --oci-worker false

sudo buildctl build --output type=image,name=myimage:latest

The daemon itself could be socket-activated, the same that is planned for docker and podman.

[afbjorklund]

Ubuntu focal amd64 package available here: https://bintray.com/afbjorklund/buildkit

Includes /usr/bin/buildctl and /usr/sbin/buildkitd, needs runc/containerd.

[afbjorklund]

It seems like the support for ssh hosts is not fully implemented in buildctl yet.

(It would have used something like: ssh USER@HOST -- buildctl dial-stdio)

But it is also possible to tunnel the unix socket over ssh explicitly, meanwhile.

ssh -nNT -L ./buildkitd.sock:/run/buildkit/buildkitd.sock USER@HOST

[afbjorklund]

See moby/buildkit#1025 for remote building HOWTO

Also KubeCon + CloudNativeCon China 2019, about
BuildKit: A Modern Builder Toolkit on Top of containerd

There is some info about buildah here: buildah_multi_stage.sh

[afbjorklund]

So far I have only done basic packaging of the binaries.

It needs to have for instance a systemd service as well

[afbjorklund]

We made sure that buildkit support was included in docker since #4143

DOCKER_BUILDKIT=1 docker build

And decided to not include buildah support (beyond "bud") in #4166

podman build ~ buildah bud (only difference is --layers)

[afbjorklund]

The binaries (buildctl and buildkitd) are now included with minikube.

However, user will have to do all the ssh commands themselves.

[afbjorklund]

Example session:

1. Start the buildkit daemon, using minikube ssh -- sudo -b buildkitd

2. Change the group of /run/buildkitd and /run/buildkit/buildkitd.sock to "docker"

3. Use minikube ssh --alsologtostderr --native-ssh=false to find out the ssh flags

4. Start a ssh tunnel from the host to the machine, for the unix socket

(ssh and flags) -nNT -L ./buildkitd.sock:/run/buildkit/buildkitd.sock &

5. Use this local (forwarded) unix socket, as the address for buildctl

buildctl --addr=unix://./buildkitd.sock build --frontend=dockerfile.v0

With a local Dockerfile, also needs: --local context=. --local dockerfile=.

For more information, see: https://github.com/moby/buildkit

---

No, this is not very user friendly (i.e. buildctl). Then again, neither is using ctr.

But the suggestion is to have other commands, such as minikube build or minikube cache

These are low-level commands, and as such they have low-level interfaces. Like crictl ?

There is some ongoing work, in order to "fix" this: https://github.com/AkihiroSuda/nerdctl