Push access is denied, even though push access is 100% available

[theAkito]

Given this Docker project:
https://github.com/theAkito/kubedigger/tree/f7bdf68171d069fd834185dbe2e1521456ac998d

Building the images resulted in the following behaviour:

[+] Building 993.7s (16/16) FINISHED                                                                                                                             
 => [internal] load .dockerignore                                                                                                                           0.0s
 => => transferring context: 2B                                                                                                                             0.0s
 => [internal] load build definition from Dockerfile                                                                                                        0.0s
 => => transferring dockerfile: 32B                                                                                                                         0.0s
 => [linux/amd64 internal] load metadata for docker.io/library/debian:testing                                                                               1.5s
 => [linux/arm64 internal] load metadata for docker.io/library/debian:testing                                                                               2.2s
 => [linux/arm/v7 internal] load metadata for docker.io/library/debian:testing                                                                              1.4s
 => [linux/arm/v7 1/3] FROM docker.io/library/debian:testing@sha256:608ee7989e7248c50c3b74eb68481f92205c8b54704eb1200d370bb3ca1a4425                        0.0s
 => => resolve docker.io/library/debian:testing@sha256:608ee7989e7248c50c3b74eb68481f92205c8b54704eb1200d370bb3ca1a4425                                     0.0s
 => [internal] load build context                                                                                                                           0.0s
 => => transferring context: 67B                                                                                                                            0.0s
 => [linux/amd64 1/3] FROM docker.io/library/debian:testing@sha256:608ee7989e7248c50c3b74eb68481f92205c8b54704eb1200d370bb3ca1a4425                         0.0s
 => => resolve docker.io/library/debian:testing@sha256:608ee7989e7248c50c3b74eb68481f92205c8b54704eb1200d370bb3ca1a4425                                     0.0s
 => [linux/arm64 1/3] FROM docker.io/library/debian:testing@sha256:608ee7989e7248c50c3b74eb68481f92205c8b54704eb1200d370bb3ca1a4425                         0.0s
 => => resolve docker.io/library/debian:testing@sha256:608ee7989e7248c50c3b74eb68481f92205c8b54704eb1200d370bb3ca1a4425                                     0.0s
 => CACHED [linux/arm64 2/3] COPY docker/bash_aliases /etc                                                                                                  0.0s
 => CACHED [linux/amd64 2/3] COPY docker/bash_aliases /etc                                                                                                  0.0s
 => CACHED [linux/arm/v7 2/3] COPY docker/bash_aliases /etc                                                                                                 0.0s
 => [linux/arm64 3/3] RUN apt-get update && apt-get install -y                            iputils-arping                                                  252.6s
 => [linux/amd64 3/3] RUN apt-get update && apt-get install -y                            iputils-arping                                                  112.9s
 => [linux/arm/v7 3/3] RUN apt-get update && apt-get install -y                            iputils-arping                                                 250.5s
 => ERROR exporting to image                                                                                                                              738.9s
 => => exporting layers                                                                                                                                    12.3s
 => => exporting manifest sha256:14d8a23b367041917084b5397bbb34c670eee026077a4d3c722d000edba6d987                                                           0.0s 
 => => exporting config sha256:3b21d435e68449096cd692272ca21a0190554dde919a391a41c2d4e4598368cb                                                             0.0s 
 => => exporting manifest sha256:8767c5a8a16bc7ecbef42b8c5e495007db0bda818a3c742ffb03a35e010c0824                                                           0.0s 
 => => exporting config sha256:95ff0e3f3fdf954de8460b07b08e82f2d737573d22a186ec5d8c43afd980b9ae                                                             0.0s 
 => => exporting manifest sha256:12fb8205526c36759b9f33b3a164fff4dbce77420e7e2c59631f6af0c9e26761                                                           0.0s 
 => => exporting config sha256:22ff1959313b8fd4e39b4c62c76bb1e8545ff9a30395a955f212e335301243ca                                                             0.0s 
 => => exporting manifest list sha256:d7b2e4949bae54acf3f5a1e87f6dc7caec7213f4746233d23e0da7e304d3454e                                                      0.0s 
 => => pushing layers                                                                                                                                     725.0s 
 => => pushing manifest for docker.io/akito13/kubedigger:0.5.0                                                                                              1.4s 
------                                                                                                                                                           
 > exporting to image:                                                                                                                                           
------                                                                                                                                                           
failed to solve: rpc error: code = Unknown desc = failed commit on ref "manifest-sha256:8767c5a8a16bc7ecbef42b8c5e495007db0bda818a3c742ffb03a35e010c0824": unexpected status: 401 Unauthorized                                                                                                                                    
[+] Building 15.0s (16/16) FINISHED                                                                                                                              
 => [internal] load build definition from Dockerfile                                                                                                        0.1s 
 => => transferring dockerfile: 32B                                                                                                                         0.0s
 => [internal] load .dockerignore                                                                                                                           0.1s
 => => transferring context: 2B                                                                                                                             0.0s
 => [linux/arm/v7 internal] load metadata for docker.io/library/debian:testing                                                                              5.1s
 => [linux/arm64 internal] load metadata for docker.io/library/debian:testing                                                                               2.1s
 => [linux/amd64 internal] load metadata for docker.io/library/debian:testing                                                                               2.1s
 => [linux/arm/v7 1/3] FROM docker.io/library/debian:testing@sha256:608ee7989e7248c50c3b74eb68481f92205c8b54704eb1200d370bb3ca1a4425                        0.0s
 => => resolve docker.io/library/debian:testing@sha256:608ee7989e7248c50c3b74eb68481f92205c8b54704eb1200d370bb3ca1a4425                                     0.0s
 => [linux/arm64 1/3] FROM docker.io/library/debian:testing@sha256:608ee7989e7248c50c3b74eb68481f92205c8b54704eb1200d370bb3ca1a4425                         0.0s
 => => resolve docker.io/library/debian:testing@sha256:608ee7989e7248c50c3b74eb68481f92205c8b54704eb1200d370bb3ca1a4425                                     0.0s
 => [linux/amd64 1/3] FROM docker.io/library/debian:testing@sha256:608ee7989e7248c50c3b74eb68481f92205c8b54704eb1200d370bb3ca1a4425                         0.0s
 => => resolve docker.io/library/debian:testing@sha256:608ee7989e7248c50c3b74eb68481f92205c8b54704eb1200d370bb3ca1a4425                                     0.0s
 => [internal] load build context                                                                                                                           0.0s
 => => transferring context: 67B                                                                                                                            0.0s
 => CACHED [linux/arm/v7 2/3] COPY docker/bash_aliases /etc                                                                                                 0.0s
 => CACHED [linux/arm/v7 3/3] RUN apt-get update && apt-get install -y                            iputils-arping                                            0.0s
 => CACHED [linux/arm64 2/3] COPY docker/bash_aliases /etc                                                                                                  0.0s
 => CACHED [linux/arm64 3/3] RUN apt-get update && apt-get install -y                            iputils-arping                                             0.0s
 => CACHED [linux/amd64 2/3] COPY docker/bash_aliases /etc                                                                                                  0.0s
 => CACHED [linux/amd64 3/3] RUN apt-get update && apt-get install -y                            iputils-arping                                             0.0s
 => exporting to image                                                                                                                                      9.8s
 => => exporting layers                                                                                                                                     0.0s
 => => exporting manifest sha256:14d8a23b367041917084b5397bbb34c670eee026077a4d3c722d000edba6d987                                                           0.0s
 => => exporting config sha256:3b21d435e68449096cd692272ca21a0190554dde919a391a41c2d4e4598368cb                                                             0.0s
 => => exporting manifest sha256:8767c5a8a16bc7ecbef42b8c5e495007db0bda818a3c742ffb03a35e010c0824                                                           0.0s
 => => exporting config sha256:95ff0e3f3fdf954de8460b07b08e82f2d737573d22a186ec5d8c43afd980b9ae                                                             0.0s
 => => exporting manifest sha256:12fb8205526c36759b9f33b3a164fff4dbce77420e7e2c59631f6af0c9e26761                                                           0.0s
 => => exporting config sha256:22ff1959313b8fd4e39b4c62c76bb1e8545ff9a30395a955f212e335301243ca                                                             0.0s
 => => exporting manifest list sha256:d7b2e4949bae54acf3f5a1e87f6dc7caec7213f4746233d23e0da7e304d3454e                                                      0.0s
 => => pushing layers                                                                                                                                       2.8s
 => => pushing manifest for docker.io/akito13/kubedigger:latest

How is it possible to get 401 Unauthorized on an empty repository, while pushing the exact same images, except with different tags (0.5.0 -> latest) again and it works?

If you look at the build script and see the buildx commands at the bottom, you see that it just uses the cached images and uploads them, basically repeating the same operation that failed before, but now resulting positively.

Not sure if these issues are related:
moby/moby#39019
#1142
#1271

[tonistiigi]

Please test with #1636 --driver-opt image=moby/buildkit:master . If you still see issues try to enable debug logs or even better if you can use the opentracing support (or a mitm proxy) to see what request is causing the failure and with what circumstances.

[theAkito]

I did:

docker buildx use "$(docker buildx create --driver-opt image=moby/buildkit:master)"

Then I ran the build script again, with the --no-cache option enabled.

This was the result:

[+] Building 718.4s (17/17) FINISHED                                                                                                                             
 => [internal] booting buildkit                                                                                                                             3.9s
 => => pulling image moby/buildkit:master                                                                                                                   2.8s
 => => creating container buildx_buildkit_fervent_tesla0                                                                                                    1.1s
 => [internal] load build definition from Dockerfile                                                                                                        0.1s
 => => transferring dockerfile: 3.22kB                                                                                                                      0.0s
 => [internal] load .dockerignore                                                                                                                           0.1s
 => => transferring context: 2B                                                                                                                             0.0s
 => [linux/arm/v7 internal] load metadata for docker.io/library/debian:testing                                                                              3.3s
 => [linux/arm64 internal] load metadata for docker.io/library/debian:testing                                                                               3.4s
 => [linux/amd64 internal] load metadata for docker.io/library/debian:testing                                                                               2.7s
 => [linux/arm/v7 1/3] FROM docker.io/library/debian:testing@sha256:608ee7989e7248c50c3b74eb68481f92205c8b54704eb1200d370bb3ca1a4425                       33.7s
 => => resolve docker.io/library/debian:testing@sha256:608ee7989e7248c50c3b74eb68481f92205c8b54704eb1200d370bb3ca1a4425                                     0.1s
 => => sha256:67f8b669b5049480f56c6fcad559f7090c9d5c691076d2136b7a479b7d84ef2b 47.62MB / 47.62MB                                                           32.1s
 => => extracting sha256:67f8b669b5049480f56c6fcad559f7090c9d5c691076d2136b7a479b7d84ef2b                                                                   1.5s
 => [internal] load build context                                                                                                                           0.3s
 => => transferring context: 1.95kB                                                                                                                         0.0s
 => [linux/arm64 1/3] FROM docker.io/library/debian:testing@sha256:608ee7989e7248c50c3b74eb68481f92205c8b54704eb1200d370bb3ca1a4425                        35.4s
 => => resolve docker.io/library/debian:testing@sha256:608ee7989e7248c50c3b74eb68481f92205c8b54704eb1200d370bb3ca1a4425                                     0.1s
 => => sha256:c3a3b210b20418d18407956d6d337463c452a063f31e10abcc972012479278c3 50.83MB / 50.83MB                                                           33.5s
 => => extracting sha256:c3a3b210b20418d18407956d6d337463c452a063f31e10abcc972012479278c3                                                                   1.9s
 => [linux/amd64 1/3] FROM docker.io/library/debian:testing@sha256:608ee7989e7248c50c3b74eb68481f92205c8b54704eb1200d370bb3ca1a4425                        14.7s
 => => resolve docker.io/library/debian:testing@sha256:608ee7989e7248c50c3b74eb68481f92205c8b54704eb1200d370bb3ca1a4425                                     0.1s
 => => sha256:4363cc52203477cd66948034ae4a1db71cbfd27fddb648dd9c590161de1f8634 51.91MB / 51.91MB                                                           13.0s
 => => extracting sha256:4363cc52203477cd66948034ae4a1db71cbfd27fddb648dd9c590161de1f8634                                                                   1.6s
 => [linux/arm/v7 2/3] COPY docker/bash_aliases /etc                                                                                                       34.0s
 => [linux/arm64 2/3] COPY docker/bash_aliases /etc                                                                                                        35.5s
 => [linux/amd64 2/3] COPY docker/bash_aliases /etc                                                                                                        15.0s
 => [linux/amd64 3/3] RUN apt-get update && apt-get install -y                            iputils-arping                                                i  49.1s
 => [linux/arm/v7 3/3] RUN apt-get update && apt-get install -y                            iputils-arping                                                 165.8s
 => [linux/arm64 3/3] RUN apt-get update && apt-get install -y                            iputils-arping                                                  169.5s
 => ERROR exporting to image                                                                                                                              505.5s
 => => exporting layers                                                                                                                                    12.2s
 => => exporting manifest sha256:59e2e7db6a26b20e4fe65c626c3d12120f9ffbdab5137ccce18700f7857f0c6e                                                           0.0s
 => => exporting config sha256:3d1b0fb26677c9883a43077076183076d8e918b61d2a9bcfb7a3b61e9fbf8446                                                             0.0s
 => => exporting manifest sha256:3bdb5ec20a11b365a4bf21e4843db83417e18e58b8a3894df2ea66ee0a2af094                                                           0.0s
 => => exporting config sha256:cc3686bb5b9a03692812dbf965db54054ee355a6504428c5245ed3c94e6390eb                                                             0.0s
 => => exporting manifest sha256:4ad1473012ca20cbe85ffd47468dcea92eaf18b6a9653fd1c2f2923d4a1a1142                                                           0.0s
 => => exporting config sha256:09b7f82b7c19d5322de4edc1ab910c6e5eb3bc60a94867f233cf975d8c34658f                                                             0.0s
 => => exporting manifest list sha256:c1e7d43df41f178b173be4266eb1abd836aea0135a24dc6bc9aad445c54b06b6                                                      0.0s
 => => pushing layers                                                                                                                                     493.2s
------
 > exporting to image:
------
failed to solve: rpc error: code = Unknown desc = failed to copy: failed to do request: Put https://registry-1.docker.io/v2/akito13/kubedigger/blobs/uploads/...: write tcp 172.17.0.3:38992->52.4.20.24:443: use of closed network connection

[theAkito]

The last one might've been an unpleasant fluke or there have been further updates to moby/buildkit:master. Seems like this issue is gone. Thanks for the help.

[dougdonohoe]

This is currently an issue on GCP Cloud Build pushing to GCP Artifact Registry when the build is longer than an hour. I filed docker/buildx#1205 with the details.

There is a workaround (build without --push, stop builder, build again). Full details with reproducible example can be found at dougdonohoe/build-timeout.