Is --config ignored when using the k8s driver

[goffinf]

re: comment from the Buildkit repo: moby/buildkit#2044 (comment)

I noted the above comment under another issue and wanted to check whether using --config with the buildx k8s driver is NOT supported right now (and if not, if it is likely to be in the near term future) ?

This is what I have observed in trying to get this to work.

I have been struggling trying to get buildx to allow me to use a private registry. Per the documentation I added --config with buildx create ..., like this:

docker buildx create --name buildx-test --config config.toml --driver kubernetes --driver-opt replicas=1,namespace=buildx --use

where the config.toml file exists in the directory where I execute the above command and contains:

debug = true
root = "/var/lib/buildkit"
insecure-entitlements = [ "network.host", "security.insecure" ]
[registry."host.k3d.internal:5000"]
http = true
insecure = true

When I request a build and use --load or --push I get the following output, which shows that the build worked successfully but buildx attempts to connect to the private registry with HTTPS rather than HTTP:

docker buildx build --platform "linux/amd64" -t "goffinf/buildx-test:1.0.0" -t "goffinf/buildx-test:latest" --build-arg [email protected] --load .

[+] Building 4.2s (7/7) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                                        0.6s
 => => transferring dockerfile: 32B                                                                                                                                                                         0.0s
 => [internal] load .dockerignore                                                                                                                                                                           0.8s
 => => transferring context: 2B                                                                                                                                                                             0.0s
 => [internal] load metadata for docker.io/library/alpine:latest                                                                                                                                            1.2s
 => [1/2] FROM docker.io/library/alpine:latest@sha256:69e70a79f2d41ab5d637de98c1e0b055206ba40a8145e7bddb55ccc04e13cf8f                                                                                      0.5s
 => => resolve docker.io/library/alpine:latest@sha256:69e70a79f2d41ab5d637de98c1e0b055206ba40a8145e7bddb55ccc04e13cf8f                                                                                      0.4s
 => CACHED [2/2] RUN mkdir /buildx &&     echo "${MESSAGE}" >> /buildx/README.md;                                                                                                                           0.0s
 => exporting to oci image format                                                                                                                                                                           1.3s
 => => exporting layers                                                                                                                                                                                     0.0s
 => => exporting manifest sha256:a8cd3a8da4fea6fa7b4ae350dd402f8274e2de864f9bd175b109d1727316aa44                                                                                                           0.3s
 => => exporting config sha256:0a31360539ed3c0e8a5c5b443d2b3fd476cc32b8967460cd38d54105888e4f3c                                                                                                             0.4s
 => => sending tarball                                                                                                                                                                                      0.4s
 => importing to docker                                                                                                                                                                                     0.1s
error: failed to do request: Head https://host.k3d.internal:5000/v2/goffinf/buildx-test/manifests/1.0.0: http: server gave HTTP response to HTTPS client

If I check the buildx container, there is definitely no sign of the toml, and the /etc/buildkit directory doesn't exist (assuming thats where it would be copied).

I can confirm that the buildx container can resolve the registry by exec'ing into the container and calling the registry API:

kubectl exec -it -n buildx buildx-test0-87c44c79c-xw4xr -- /bin/sh

/ # wget http://host.k3d.internal:5000/v2/_catalog
Connecting to host.k3d.internal:5000 (172.19.0.1:5000)
saving to '_catalog'
_catalog             100% |*****************************************************************************************************************************************************************|   280  0:00:00 ETA
'_catalog' saved

/ # cat _catalog
{"repositories":["alpine","busybox","docker","docker/compose","goffinf/azure-agent","goffinf/azure-agent-standalone","goffinf/build-test","goffinf/k8s-tools","goffinf/rancher-tools","goffinf/rvault","goffinf/terraform","kir4h/rvault","nginx","nginxdemos/hello","ubuntu","vault"]}

Kind Regards

Fraser Goffin

[tonistiigi]

@AkihiroSuda @morlay

[AkihiroSuda]

This is unimplemented yet

[goffinf]

@AkihiroSuda , thanks for confirming. Can I ask whether there are any plans to implement this in the near term, or whether you need a feature request to be raised ?

Can you suggest any alternate approach to make use of a private registry in the mean time (we wish to maintain a consistent implementation across our development, staging and release environments iro registry integration - public registries are not permitted by our CISO)?

Thanks.

[AkihiroSuda]

PR is wanted. A workaround is to build your own moby/buildkit image with the config file, and specify the image with --driver-opt image=IMAGE.

[goffinf]

Thanks @AkihiroSuda I thought that (workaround) would be needed for now.

I guess a PR would be appropriate once a candidate implementation has been developed. I suspect that is beyond my talents (although I am will to try if you want to point me in the right direction).

Regards.

[goffinf]

Some additional information which may be helpful.

As suggested, using a custom image which includes a buildkitd.toml in the /etc/buildkit directory and referencing that image when calling docker buildx create ... does allow a successful build and load (or push):

docker buildx create --name buildx-test --driver kubernetes --driver-opt image=goffinf/buildkit:buildx-stable-1,replicas=1,namespace=buildx --use;
docker buildx inspect --bootstrap;
...
docker buildx build --platform "linux/amd64" -t host.k3d.internal:5000/goffinf/buildx-test:1.0.0 -t host.k3d.internal:5000/goffinf/buildx-test:latest --load .
[+] Building 4.3s (7/7) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                                        0.7s
 => => transferring dockerfile: 32B                                                                                                                                                                         0.0s
 => [internal] load .dockerignore                                                                                                                                                                           1.0s
 => => transferring context: 2B                                                                                                                                                                             0.0s
 => [internal] load metadata for docker.io/library/alpine:latest                                                                                                                                            1.1s
 => [1/2] FROM docker.io/library/alpine:latest@sha256:69e70a79f2d41ab5d637de98c1e0b055206ba40a8145e7bddb55ccc04e13cf8f                                                                                      0.4s
 => => resolve docker.io/library/alpine:latest@sha256:69e70a79f2d41ab5d637de98c1e0b055206ba40a8145e7bddb55ccc04e13cf8f                                                                                      0.4s
 => CACHED [2/2] RUN mkdir /buildx &&     echo "${MESSAGE}" >> /buildx/README.md;                                                                                                                           0.0s
 => exporting to oci image format                                                                                                                                                                           1.4s
 => => exporting layers                                                                                                                                                                                     0.0s
 => => exporting manifest sha256:c5498a8885541a89c526c56eae7adb8f2fa80710c18d5819bd2bd218f352a47d                                                                                                           0.4s
 => => exporting config sha256:df3f4f75e6e0a85a324f600c828411c19a03e9ac2ff006608a3222fe79aa9414                                                                                                             0.4s
 => => sending tarball                                                                                                                                                                                      0.4s
 => importing to docker

--push also works fine.

However, ....

If I then attempt to execute docker buildx imagetools inspect my-private-registry/image:tag then I get the original error:

docker buildx imagetools inspect host.k3d.internal:5000/goffinf/buildkit-test:v1.0.0

error: failed to do request: Head https://host.k3d.internal:5000/v2/goffinf/buildkit-test/manifests/v1.0.0: http: server gave HTTP response to HTTPS client

Using a docker.io image is ok:

docker buildx imagetools inspect alpine:latest
Name:      docker.io/library/alpine:latest
MediaType: application/vnd.docker.distribution.manifest.list.v2+json
Digest:    sha256:69e70a79f2d41ab5d637de98c1e0b055206ba40a8145e7bddb55ccc04e13cf8f

Manifests:
  Name:      docker.io/library/alpine:latest@sha256:def822f9851ca422481ec6fee59a9966f12b351c62ccb9aca841526ffaa9f748
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/amd64
...