Pushing an image to GCR under a new tag fails with 401 Unauthorized

[howardjohn]

$ echo 'FROM alpine' > Dockerfile
$ docker buildx build . -t gcr.io/.../cache-test:latest --push
[+] Building 5.4s (5/5) FINISHED
 => [internal] load .dockerignore                                                                                                  0.0s
 => => transferring context: 2B                                                                                                    0.0s
 => [internal] load build definition from Dockerfile                                                                               0.0s
 => => transferring dockerfile: 49B                                                                                                0.0s
 => [internal] load metadata for docker.io/library/alpine:latest                                                                   1.3s
 => CACHED [1/1] FROM docker.io/library/alpine@sha256:c0e9560cda118f9ec63ddefb4a173a2b2a0347082d7dff7dc14272e7841a5b5a             0.0s
 => => resolve docker.io/library/alpine@sha256:c0e9560cda118f9ec63ddefb4a173a2b2a0347082d7dff7dc14272e7841a5b5a                    0.0s
 => exporting to image                                                                                                             4.1s
 => => exporting layers                                                                                                            0.0s
 => => exporting manifest sha256:4567b80f4b7d8f139e9574aaa0850c8c4d89cc36b40f973902eab249492a78c6                                  0.0s
 => => exporting config sha256:8d19be48f0403c567476dc8607fc05eeb8a33846798029f7b4bd2e94f3ee331f                                    0.0s
 => => pushing layers                                                                                                              2.9s
 => => pushing manifest for gcr.io/<project>/cache-test:latest                                                              1.1s
$ docker buildx build . -t gcr.io/<project>/cache-test:v1 --push
[+] Building 2.3s (5/5) FINISHED
 => [internal] load .dockerignore                                                                                                  0.0s
 => => transferring context: 2B                                                                                                    0.0s
 => [internal] load build definition from Dockerfile                                                                               0.0s
 => => transferring dockerfile: 31B                                                                                                0.0s
 => [internal] load metadata for docker.io/library/alpine:latest                                                                   1.3s
 => CACHED [1/1] FROM docker.io/library/alpine@sha256:c0e9560cda118f9ec63ddefb4a173a2b2a0347082d7dff7dc14272e7841a5b5a             0.0s
 => => resolve docker.io/library/alpine@sha256:c0e9560cda118f9ec63ddefb4a173a2b2a0347082d7dff7dc14272e7841a5b5a                    0.0s
 => ERROR exporting to image                                                                                                       1.0s
 => => exporting layers                                                                                                            0.0s
 => => exporting manifest sha256:4567b80f4b7d8f139e9574aaa0850c8c4d89cc36b40f973902eab249492a78c6                                  0.0s
 => => exporting config sha256:8d19be48f0403c567476dc8607fc05eeb8a33846798029f7b4bd2e94f3ee331f                                    0.0s
 => => pushing layers                                                                                                              0.2s
 => => pushing manifest for gcr.io/<project>/cache-test:v1                                                                  0.7s
------
 > exporting to image:
------
failed to solve: rpc error: code = Unknown desc = failed commit on ref "manifest-sha256:4567b80f4b7d8f139e9574aaa0850c8c4d89cc36b40f973902eab249492a78c6": unexpected status: 401 Unauthorized

The same commands when used in a non-buildx context work fine

$ docker version
Client: Docker Engine - Community
 Version:           20.10.0-rc1
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        5cc2396
 Built:             Tue Nov 17 22:48:41 2020
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.0-rc1
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       131bf7e
  Built:            Tue Nov 17 22:52:57 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.4.1
  GitCommit:        c623d1b36f09f8ef6536a057bd658b3aa8632828
 runc:
  Version:          1.0.0-rc92
  GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

$ uname -a
Linux howardjohn-glaptop2 5.7.17-1rodete4-amd64 #1 SMP Debian 5.7.17-1rodete4 (2020-10-01) x86_64 GNU/Linux

[howardjohn]

The exact same commands with a dockerhub registry work fine

[howardjohn]

Builder logs:

/moby.buildkit.v1.Control/Solve returned error: unexpected status: 401 Unauthorized
github.com/containerd/containerd/remotes/docker.(*pushWriter).Commit
        /src/vendor/github.com/containerd/containerd/remotes/docker/pusher.go:347
github.com/containerd/containerd/content.Copy
        /src/vendor/github.com/containerd/containerd/content/helpers.go:151
github.com/containerd/containerd/remotes.push
        /src/vendor/github.com/containerd/containerd/remotes/handlers.go:171
github.com/containerd/containerd/remotes.PushHandler.func1
        /src/vendor/github.com/containerd/containerd/remotes/handlers.go:146
github.com/moby/buildkit/util/push.Push
        /src/util/push/push.go:103
github.com/moby/buildkit/exporter/containerimage.(*imageExporterInstance).Export
        /src/exporter/containerimage/export.go:240
github.com/moby/buildkit/solver/llbsolver.(*Solver).Solve.func3
        /src/solver/llbsolver/solver.go:218
github.com/moby/buildkit/solver/llbsolver.inVertexContext
        /src/solver/llbsolver/solver.go:359
github.com/moby/buildkit/solver/llbsolver.(*Solver).Solve
        /src/solver/llbsolver/solver.go:217
github.com/moby/buildkit/control.(*Controller).Solve
        /src/control/control.go:276
github.com/moby/buildkit/api/services/control._Control_Solve_Handler.func1
        /src/api/services/control/control.pb.go:1489
github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1
        /src/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc/server.go:57
main.unaryInterceptor.func1
        /src/cmd/buildkitd/main.go:531
github.com/moby/buildkit/api/services/control._Control_Solve_Handler
        /src/api/services/control/control.pb.go:1491
google.golang.org/grpc.(*Server).processUnaryRPC
        /src/vendor/google.golang.org/grpc/server.go:1024
google.golang.org/grpc.(*Server).handleStream
        /src/vendor/google.golang.org/grpc/server.go:1313
google.golang.org/grpc.(*Server).serveStreams.func1.1
        /src/vendor/google.golang.org/grpc/server.go:722
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1357
failed commit on ref "manifest-sha256:4567b80f4b7d8f139e9574aaa0850c8c4d89cc36b40f973902eab249492a78c6"
github.com/containerd/containerd/content.Copy
        /src/vendor/github.com/containerd/containerd/content/helpers.go:153
github.com/containerd/containerd/remotes.push
        /src/vendor/github.com/containerd/containerd/remotes/handlers.go:171
github.com/containerd/containerd/remotes.PushHandler.func1
        /src/vendor/github.com/containerd/containerd/remotes/handlers.go:146
github.com/moby/buildkit/util/push.Push
        /src/util/push/push.go:103
github.com/moby/buildkit/exporter/containerimage.(*imageExporterInstance).Export
        /src/exporter/containerimage/export.go:240
github.com/moby/buildkit/solver/llbsolver.(*Solver).Solve.func3
        /src/solver/llbsolver/solver.go:218
github.com/moby/buildkit/solver/llbsolver.inVertexContext
        /src/solver/llbsolver/solver.go:359
github.com/moby/buildkit/solver/llbsolver.(*Solver).Solve
        /src/solver/llbsolver/solver.go:217
github.com/moby/buildkit/control.(*Controller).Solve
        /src/control/control.go:276
github.com/moby/buildkit/api/services/control._Control_Solve_Handler.func1
        /src/api/services/control/control.pb.go:1489
github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1
        /src/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc/server.go:57
main.unaryInterceptor.func1
        /src/cmd/buildkitd/main.go:531
github.com/moby/buildkit/api/services/control._Control_Solve_Handler
        /src/api/services/control/control.pb.go:1491
google.golang.org/grpc.(*Server).processUnaryRPC
        /src/vendor/google.golang.org/grpc/server.go:1024
google.golang.org/grpc.(*Server).handleStream
        /src/vendor/google.golang.org/grpc/server.go:1313
google.golang.org/grpc.(*Server).serveStreams.func1.1
        /src/vendor/google.golang.org/grpc/server.go:722
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1357

[howardjohn]

Possibly related to moby/buildkit#1143

[tonistiigi]

@howardjohn Are your conditions same as containerd/containerd#4622 ?

[howardjohn]

@tonistiigi I think this is fixed by a change in GCR - at least I cannot reproduce anymore. Since the fix was internal I suspect this was one was a different root cause, although I am not certain. Let me check to verify this is fully fixed and I will close if so