kubernetes driver ignores --driver-opt image when rootless opt is set

[ghostcoder007]

Respected members of the buildx team and community,

We are using buildx to generate multi-arch images in arm64/amd64 architectures using the kubernetes driver. We are pushing to a private registry and we have a custom base image with root certificates to access the private registry. The commands we use are:

# AMD64
docker buildx create --use --name demo-builder --node=demo-builder-amd64 --platform linux/amd64 \
  --driver kubernetes --driver-opt image=demo/demo-buildkit:amd64 --driver-opt namespace=demo-builder \
  --driver-opt replicas=1 --driver-opt nodeselector=beta.kubernetes.io/arch=amd64 --driver-opt rootless=true

# ARM64
docker buildx create --append --name demo-builder --node=demo-builder-arm64 --platform linux/arm64 \
  --driver kubernetes --driver-opt image=demo/demo-buildkit:arm64 --driver-opt namespace=demo-builder \
  --driver-opt replicas=1 --driver-opt nodeselector=beta.kubernetes.io/arch=arm64 --driver-opt rootless=true

# Now build using the buildx builder setup above
docker buildx build --builder demo-builder --platform linux/amd64,linux/arm64 -f images/demo/Dockerfile .

But intermittently we see that the --driver-opt image=demo/demo-buildkit:amd64 or --driver-opt image=demo/demo-buildkit:arm64 gets ignored and the kubernetes pod is created with moby/buildkit image:

$ k -n demo-builder describe pod demo-arm64-64f565cf56-2vssj
Name:         demo-builder-arm64-64f565cf56-2vssj
Namespace:    demo-builder
Priority:     0
Node:         ip-10-13-13-114.ec2.internal/10.13.13.114
Start Time:   Mon, 07 Feb 2022 11:48:33 +0000
Labels:       app=demo-builder-arm64
              pod-template-hash=64f565cf56
Annotations:  buildx.docker.com/platform: linux/arm64
              container.apparmor.security.beta.kubernetes.io/buildkitd: unconfined
              container.seccomp.security.alpha.kubernetes.io/buildkitd: unconfined
              kubernetes.io/psp: eks.privileged
Status:       Running
IP:           192.168.0.1
IPs:
  IP:           192.168.0.1
Controlled By:  ReplicaSet/demo-builder-arm64-64f565cf56
Containers:
  buildkitd:
    Container ID:  docker://d89d8d1e37a1836a8d9e8e5834df9f0d7c19059723f9c568fe52dc71b364ac28
    Image:         moby/buildkit:buildx-stable-1-rootless
    Image ID:      docker-pullable://moby/buildkit@sha256:d877f877f7411804fefaf90071464913130f8d34fe5797ac2f7164ddf816d27c

A weird observation is that sometimes of out of the two nodes, the option i.e. --driver-opt image=demo/demo-buildkit:amd64 is ignored for just one node and the second node works fine. And some other times both work fine and images get built correctly.

Because of this problem, our builds are extremely inconsistent and fail with the error below when moby/buildkit image is used:

error: failed to solve: docker.example.com/oraclelinux:7-slim@sha256:bb7c3969d33b3c2695b11dd705e18ed604ce0f1e3317ef293e8f0d9d125dc90a: failed to do request: Head "https://docker.example.com/v2/oraclelinux/manifests/sha256:bb7c3969d33b3c2695b11dd705e18ed604ce0f1e3317ef293e8f0d9d125dc90a": x509: certificate signed by unknown authority

Our build environment details:

$ docker version
Client:
 Cloud integration: v1.0.22
 Version:           20.10.12
 API version:       1.41
 Go version:        go1.16.12
 Git commit:        e91ed57
 Built:             Mon Dec 13 11:46:56 2021
 OS/Arch:           darwin/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.12
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.12
  Git commit:       459d0df
  Built:            Mon Dec 13 11:43:56 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.12
  GitCommit:        7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Any guidance or advice is highly appreciated! Thanks! 🙇

[ghostcoder007]

One more observation, removing --driver-opt rootless=true and passing in a image with root user works fine and is a possible workaround.

[ghostcoder007]

Another observation, this used to work fine at least till docker client version 20.10.6. So, some change between 20.10.6 and 20.10.12 should be causing it.

[tonistiigi]

Yes, rootless overrides the image property with default image atm. https://github.com/docker/buildx/blob/master/driver/kubernetes/factory.go#L109 . It should not do that when custom image is set. Atm it looks like the behavior might even be random as maps in Go are not sorted.

@morlay @AkihiroSuda

[sourikghosh]

@tonistiigi how to check if the custom image is set or not