docker buildx build hangs when using some cached layers

[ojab]

Copypaste of #2009 with different versions and backtraces.

docker-20.10.7, buildx-github.com/docker/buildx v0.5.1 11057da37336192bfc57d81e02359ba7ba848e4a (nigthly) setup via docker buildx create --name builder-builder --platform arm64,amd64 --driver docker-container --driver-opt image=moby/buildkit:nightly --buildkitd-flags '--debug' --use.

Unfortunately cannot provide a reproducer, since it's internal project.

Logs after SIGQUIT to buildkitd: https://gist.github.com/a6a9de6c613949cf6bbababef2163b7d
In Dockerfile there is

COPY --from=cloner REVISION /REVISION

ENV RAILS_SERVE_STATIC_FILES=true
ENV PORT=3002

and in the docker buildx build output last lines are

 => [linux/amd64 scheduler 12/13] COPY --from=cloner /app /app                                                                12.4s
 => [linux/arm64 scheduler 12/13] COPY --from=cloner /app /app                                                                10.8s
 => [linux/amd64 scheduler 13/13] COPY --from=cloner REVISION /REVISION                                                        5.4s
 => [linux/arm64 scheduler 13/13] COPY --from=cloner REVISION /REVISION                                                        5.4s

perf top -g -p ${buildkitd_pid} shows

  Children      Self  Shared Object     Symbol                                                                                      
+   99,78%     2,81%  buildkitd         [.] io.init
+   99,71%     7,10%  buildkitd         [.] github.com/containerd/containerd/api/services/namespaces/v1.(*GetNamespaceResponse).Stri
+   99,71%     2,87%  buildkitd         [.] github.com/containerd/containerd/api/services/namespaces/v1.(*ListNamespacesRequest).Str
+   26,38%     0,00%  buildkitd         [.] github.com/gogo/protobuf/types.(*Syntax).EnumDescriptor
+   26,38%     0,00%  buildkitd         [.] github.com/gogo/protobuf/types.init.19
+   26,38%     0,00%  buildkitd         [.] github.com/gogo/protobuf/types.(*EnumValue).Unmarshal
+   26,38%     0,00%  buildkitd         [.] github.com/moby/buildkit/cache/metadata.(*StorageItem).SetValue
+   26,38%     0,00%  buildkitd         [.] github.com/moby/buildkit/util/grpcerrors.each
+   26,38%     0,00%  buildkitd         [.] github.com/moby/buildkit/util/grpcerrors.FromGRPC
+   26,38%     0,00%  buildkitd         [.] type.*+0x1f52a9
+   26,38%     0,00%  buildkitd         [.] github.com/gogo/googleapis/google/rpc.(*DebugInfo).Equal
+   26,38%     0,00%  buildkitd         [.] github.com/moby/buildkit/client/llb.(*State).Output
+   26,38%     0,00%  buildkitd         [.] github.com/moby/buildkit/cache.migrateChainID
+   26,38%     0,00%  buildkitd         [.] github.com/containerd/containerd/runtime/v2/runc/options.(*CheckpointOptions).XXX_Merge
+   26,38%     0,00%  buildkitd         [.] github.com/containerd/containerd/plugin.Register
+   26,38%     0,00%  buildkitd         [.] github.com/google/uuid.Parse
+   26,38%     0,00%  buildkitd         [.] github.com/containerd/containerd/api/services/snapshots/v1.(*UpdateSnapshotRequest).Stri
+   26,38%     0,00%  buildkitd         [.] github.com/containerd/containerd/api/services/namespaces/v1.(*ListNamespacesResponse).Ma
+   26,38%     0,00%  buildkitd         [.] github.com/containerd/containerd/api/services/namespaces/v1.(*ListNamespacesRequest).Mar
+   17,07%    16,78%  buildkitd         [.] runtime.mapaccess1_fast32
+   14,52%     0,32%  buildkitd         [.] internal/reflectlite.maplen
+   13,48%     1,11%  buildkitd         [.] github.com/containerd/containerd/api/services/namespaces/v1.(*GetNamespaceRequest).Unmar
+   11,64%    11,41%  buildkitd         [.] runtime.typedslicecopy
+    8,27%     8,16%  buildkitd         [.] runtime.findObject
+    6,21%     6,10%  buildkitd         [.] type..eq.sync.poolLocal
+    5,69%     5,60%  buildkitd         [.] runtime.mapaccess1_faststr
+    5,56%     5,44%  buildkitd         [.] runtime.mapassign_fast32
+    3,94%     3,88%  buildkitd         [.] reflect.typedmemclrpartial
+    3,94%     3,89%  buildkitd         [.] runtime.mapaccess2_fast32
+    2,79%     2,76%  buildkitd         [.] internal/reflectlite.typedmemmove
+    2,67%     2,63%  buildkitd         [.] reflect.mapassign
+    2,46%     2,41%  buildkitd         [.] reflect.typedslicecopy
+    2,28%     2,26%  buildkitd         [.] type..eq.sync.Cond

100% reproducible with --cache-from, works fine without --cache-from.

[ojab]

And it happens often here. As a workaround we're removing tags used as a cache, but it happens in subsequent builds after that.

[tonistiigi]

Unfortunately cannot provide a reproducer, since it's internal project.

Please try to replicate the behavior on a project you can share.