Long running push to the hub end-up failing with 401 after successfully pushing the layers

[dubo-dubon-duponey]

My upload speed is pretty bad these days.

Smaller builds push to the hub: fine.

All builds push to a local registry: fine.

Bigger builds push to the hub (> 10 minutes):

• layers seem to push fine
• fails with 401 when trying to push the manifests

My hunch is that the token expired while pushing the layers, then the client fails to refresh the auth token.

(cc @dmcgowan because we discussed this a long time ago ;-))

Let me know what additional info would help.

Typical output:

 => ERROR exporting to image                                                                                                                                                                               473.1s
 => => exporting layers                                                                                                                                                                                     28.9s
 => => exporting manifest sha256:e36ae68dad79c81a00eafe72558737f6574361ac83a216874e656df388995845                                                                                                            0.0s
 => => exporting config sha256:b17d37d3620f94fa2a8aef8df2ccbfe7e3474024be2dbab7c21418106f4abe15                                                                                                              0.0s
 => => exporting manifest sha256:66c2e950c4482383066bf13a5eef3745432dfe656ce2bab7ac5e722783f4a1fd                                                                                                            0.0s
 => => exporting config sha256:bd93b437f87c69a949b5ad5e9c608f13bb7bdb9426ae42f2e96f0d6312ce6b55                                                                                                              0.0s
 => => exporting manifest sha256:730f768af200ff6ae8b89a11e13237631007ae1386fbc6be3534ad9286a5d9d8                                                                                                            0.0s
 => => exporting config sha256:8352ef1d452a9005a4f8d48f8d134ea024cb392ccdac48e6fd290b21cab586bb                                                                                                              0.0s
 => => exporting manifest sha256:813a4538a9292d498c02f1d0667c4862768304a586a3147e9c5414fb3c228eab                                                                                                            0.0s
 => => exporting config sha256:b0b27bfcb829a24c7ef555ffcb8a90df94f13f38fca03d2aa71ada711021c3c1                                                                                                              0.0s
 => => exporting manifest list sha256:d2e0997fdaf7f267444023c945c84265171331a28a1862974c0e7fef1bd1942d                                                                                                       0.0s
 => => pushing layers                                                                                                                                                                                      395.8s
 => => pushing manifest for registry.REDACTED/dubodubonduponey/goproxy:latest                                                                                                                        0.2s
 => => pushing manifest for docker.io/dubodubonduponey/goproxy:latest                                                                                                                                        1.2s
------
 > exporting to image:
------
failed to solve: rpc error: code = Unknown desc = failed commit on ref "manifest-sha256:e36ae68dad79c81a00eafe72558737f6574361ac83a216874e656df388995845": unexpected status: 401 Unauthorized

real	10m20.713s
user	0m3.316s
sys	0m6.647s

 => ERROR exporting to image                                                                                                                                                                               449.6s
 => => exporting layers                                                                                                                                                                                     25.0s
 => => exporting manifest sha256:5edda9029933eda02315f8bb4e2e600694887ac4acbc42ef005ba46df8d9fbcd                                                                                                            0.0s
 => => exporting config sha256:43242eaff830f2e1de04d26b2f843da29c1e4c2fe7b6cb9e0b281879b93f2138                                                                                                              0.0s
 => => exporting manifest sha256:44ee82caef913553faa0b3438a88300e7d547159708e4cc60d68dc4827703333                                                                                                            0.0s
 => => exporting config sha256:eb0b8123c107d71b75c76d9ef78451543b5f0601df9f7495918aec486488cf20                                                                                                              0.0s
 => => exporting manifest sha256:70783241f2180d9fd95b838db95a89f53b873ef09a77342eb15b1f9ecf2bab3d                                                                                                            0.0s
 => => exporting config sha256:238c91d8ae68f99b5889bd63fc17d2ec050ffd5286fca1b8e1101a763927b544                                                                                                              0.0s
 => => exporting manifest sha256:d1e37870e3bd86e69ceb8d197b215776d021cf366b221a210ae8d6862ef854e7                                                                                                            0.0s
 => => exporting config sha256:a95c1b68ebff56e46d952168828b66c62280a961d50bde87988ea01cbced9b39                                                                                                              0.0s
 => => exporting manifest list sha256:068c9ada19927f3a25d00ee2d2ae0c73fb9bc12bb624002081ab40fd8b7cb67b                                                                                                       0.0s
 => => pushing layers                                                                                                                                                                                      386.8s
 => => pushing manifest for registry.REDACTED/dubodubonduponey/goproxy:latest                                                                                                                        0.2s
 => => pushing manifest for docker.io/dubodubonduponey/goproxy:latest                                                                                                                                        1.1s
------
 > exporting to image:
------
failed to solve: rpc error: code = Unknown desc = failed commit on ref "manifest-sha256:70783241f2180d9fd95b838db95a89f53b873ef09a77342eb15b1f9ecf2bab3d": unexpected status: 401 Unauthorized

real	11m2.507s
user	0m3.926s
sys	0m6.978s

[blabla1337]

Similar as mine issue over here: #327
And also like this issue but closed without a proper fix: #139

[tonistiigi]

These cases should be fixed now with moby/buildkit#1636 if you can verify with the master buildkit image (you can pass on buildx create) and confirm.

[dubo-dubon-duponey]

@tonistiigi running off master now. So far, I have not hit the problem mentioned above yet.

Now, I'm getting:

failed to solve: rpc error: code = Unknown desc = failed to copy: failed to do request: Put https://registry-1.docker.io/v2/REDACTED/base/blobs/uploads/REDACTED?_state=REDACTED%3D%3D&digest=REDACTED: write tcp 172.17.0.2:59758->52.4.20.24:443: use of closed network connection

(Not the first time I see this btw ^)

[dubo-dubon-duponey]

@tonistiigi I've been pushing quite a few big (multi-arch) images to the hub now without hitting the original issue anymore (some pushes have ran way over 30 minutes).

I'm going to close since it appears resolved.
If I still hit the "use of closed network connection" error I will open a follow-up ticket.

Thanks a lot for the work on this!!!

[dougdonohoe]

This is currently an issue on GCP Cloud Build pushing to GCP Artifact Registry when the build is longer than an hour. I filed #1205 with the details.

There is a workaround (build without --push, stop builder, build again). Full details with reproducible example can be found at dougdonohoe/build-timeout.