buildx create kubernetes driver --config flag not working

[colachg]

I have searched a lot in this repo and find a related issue #136 (comment)

Because of dockerhub's pull limit, I have to use a mirror to do daily build. I decide to use buildx k8s as my build instance and follow this doc, but I find it did pull from registry-1.docker.io not my expected mirror.
Here is my custom buildkitd.toml:

buildkitd.toml:

debug = true
# root is where all buildkit state is stored.
root = "/var/lib/buildkit"
# insecure-entitlements allows insecure entitlements, disabled by default.
insecure-entitlements = [ "network.host", "security.insecure" ]

# registry configures a new Docker register used for cache import or output.
[registry."docker.io"]
  mirrors = ["mirror.ccs.tencentyun.com"]
  http = true
  insecure = false

After I use docker buildx create --name builder --node amd64 --platform=linux/amd64 --driver kubernetes --driver-opt image=moby/buildkit:master,namespace=builder,replicas=3,requests.cpu=4,requests.memory=8G,limits.cpu=4,limits.memory=8G,nodeselector=kubernetes.io/arch=amd64 --config buildkitd.toml;
to create a buildx kubernetes instance. I can see the instance:

[root@master1 ~]# kubectl -nbuilder get po -owide
NAME                     READY   STATUS    RESTARTS   AGE    IP            NODE         NOMINATED NODE   READINESS GATES
amd64-6898f948fc-frvz8   1/1     Running   0          20h    10.13.4.141   10.0.99.11   <none>           <none>
arm64-6879fcd9bf-4cp4n   1/1     Running   0          100m   10.13.3.150   10.0.99.24   <none>           <none>

Here is my test Dockerfike:

FROM alpine
RUN apk add --update curl

I test with

docker buildx build . --platform=linux/amd64,linux/arm64 --builder builder --push -t demo.goharbor.io/library/alpine:`date +"%Y%m%d"`

At the meantime I capture the traffic on 10.0.99.11 and 10.0.99.24 with the cmd: tcpdum -i any tcp and host registry-1.docker.io . I can see the traffic to this site. And I find nothing about the buildkitd.toml inside the pod and deployment details.

It's weird so I think if I modify the deployment of my buildx instance and manually mount the buildkitd.toml to the pod, will it works?

Here is what I do:

# create configmap of buildkitd.toml
kubectl -nbuilder create configmap buildkitd.toml --from-file=buildkitd.toml 

buildkitd-patch.yaml:

---
spec:
  template:
    spec:
      containers:
        -
          image: "moby/buildkit:master"
          name: buildkitd
          volumeMounts:
            -
              mountPath: /etc/buildkit/
              name: buildkit-cm
      volumes:
        -
          configMap:
            defaultMode: 420
            items:
              -
                key: buildkitd.toml
                path: buildkitd.toml
            name: buildkitd.toml
          name: buildkit-cm

kubectl -nbuilder patch deployment arm64 --patch "$(cat buildkitd-patch.yaml)"
kubectl -nbuilder patch deployment amd64 --patch "$(cat buildkitd-patch.yaml)"

Finally I test with building with buildx and capturing traffic on both mirror site and dockerhub site. It works as expected that pulls from mirror.

Maybe it's a workaround @tangx cc @tonistiigi

[morlay]

PR ready #682, but not merged

[AkihiroSuda]

#682 is now merged, thanks for working on this