Multiple targets bake is slow to send context even if all targets use the same.

[dgageot]

I've got a bake build that builds around 20 targets concurrently.
All of the targets share the same context, a Go project of around 100Mb.
When I run buildx bake, it's visible that it sends the same context to the buildkit, for each target.
I thought it was the logs that made me believe it was so but that underneath it was all fine and sent only once.

So I created a "minimal" test project with 40 targets building the same
Go sources. I added a 100Mb useless file to the context.

docker buildx bake -f slow.hcl --load takes multiple minutes to complete and we can see that the context is being sent 40 times.

I tested another option where I add a base target to the bakefile, copy the whole context into that image and made the Dockerfile copy from this target instead of directly from .
Now, the build is way less verbose and finished in 19s with:

BASE="target:base" docker buildx bake -f slow.hcl --load

By the way, this workaround requires changing all my Dockerfiles. Every COPY x y becomes COPY --from=base x y. It would be much simpler if we could redefine the "." context and set it to "target:base" using build-contexts flags.

Another thing interesting is how the build takes 19s and if you sum up the build logs, you've got 4.7s.

[+] Building 18.5s (173/173) FINISHED
 => [base internal] load build definition from Dockerfile                                                            0.0s
 => => transferring dockerfile: 59B                                                                                  0.0s
 => [base internal] load .dockerignore                                                                               0.0s
 => => transferring context: 2B                                                                                      0.0s
 => [pkg26 internal] load build context                                                                              0.0s
 => => transferring context: 7.50kB                                                                                  0.0s
 => CACHED [pkg26 1/1] COPY . .                                                                                      0.1s
 => [pkg26] exporting to image                                                                                       0.7s
 => => exporting layers                                                                                              0.0s
 => => writing image sha256:b028daff45f597ec1bf4a6a090391c5dce0fff854def9057b4ee005ee79ff8ea                         0.0s
 => => writing image sha256:82f0fd8aa9e650107ba5a6909752e26afe17d183af92d54ec28396ad44af70e5                         0.0s
 => [pkg04 internal] load build definition from Dockerfile                                                           0.0s
 => => transferring dockerfile: 360B                                                                                 0.0s
 => [pkg12 internal] load build definition from Dockerfile                                                           0.0s
 => => transferring dockerfile: 32B                                                                                  0.0s
 => [pkg32 internal] load build definition from Dockerfile                                                           0.0s
 => => transferring dockerfile: 32B                                                                                  0.0s
 => [pkg30 internal] load build definition from Dockerfile                                                           0.0s
 => => transferring dockerfile: 32B                                                                                  0.0s
 => [pkg25 internal] load build definition from Dockerfile                                                           0.0s
 => => transferring dockerfile: 32B                                                                                  0.0s
 => [pkg05 internal] load build definition from Dockerfile                                                           0.0s
 => => transferring dockerfile: 32B                                                                                  0.0s
 => [pkg18 internal] load build definition from Dockerfile                                                           0.0s
 => => transferring dockerfile: 32B                                                                                  0.0s
 => [pkg14 internal] load build definition from Dockerfile                                                           0.0s
 => => transferring dockerfile: 32B                                                                                  0.0s
 => [pkg15 internal] load build definition from Dockerfile                                                           0.0s
 => => transferring dockerfile: 32B                                                                                  0.0s
 => [pkg02 internal] load build definition from Dockerfile                                                           0.0s
 => => transferring dockerfile: 32B                                                                                  0.0s
 => [pkg10 internal] load build definition from Dockerfile                                                           0.0s
 => => transferring dockerfile: 32B                                                                                  0.0s
 => [pkg38 internal] load build definition from Dockerfile                                                           0.0s
 => => transferring dockerfile: 32B                                                                                  0.0s
 => [pkg36 internal] load build definition from Dockerfile                                                           0.0s
 => => transferring dockerfile: 32B                                                                                  0.0s
 => [pkg19 internal] load build definition from Dockerfile                                                           0.0s
 => => transferring dockerfile: 32B                                                                                  0.0s
 => [pkg23 internal] load build definition from Dockerfile                                                           0.0s
 => => transferring dockerfile: 32B                                                                                  0.0s
 => [pkg39 internal] load build definition from Dockerfile                                                           0.0s
 => => transferring dockerfile: 32B                                                                                  0.0s
 => [pkg03 internal] load build definition from Dockerfile                                                           0.0s
 => => transferring dockerfile: 32B                                                                                  0.0s
 => [pkg28 internal] load build definition from Dockerfile                                                           0.0s
 => => transferring dockerfile: 32B                                                                                  0.0s
 => [pkg08 internal] load build definition from Dockerfile                                                           0.0s
 => => transferring dockerfile: 32B                                                                                  0.0s
 => [pkg37 internal] load build definition from Dockerfile                                                           0.0s
 => => transferring dockerfile: 32B                                                                                  0.0s
 => [pkg21 internal] load build definition from Dockerfile                                                           0.0s
 => => transferring dockerfile: 32B                                                                                  0.0s
 => [pkg35 internal] load build definition from Dockerfile                                                           0.0s
 => => transferring dockerfile: 32B                                                                                  0.0s
 => [pkg20 internal] load build definition from Dockerfile                                                           0.0s
 => => transferring dockerfile: 32B                                                                                  0.0s
 => [pkg06 internal] load build definition from Dockerfile                                                           0.0s
 => => transferring dockerfile: 32B                                                                                  0.0s
 => [pkg00 internal] load build definition from Dockerfile                                                           0.0s
 => => transferring dockerfile: 32B                                                                                  0.0s
 => [pkg13 internal] load build definition from Dockerfile                                                           0.0s
 => => transferring dockerfile: 32B                                                                                  0.0s
 => [pkg33 internal] load build definition from Dockerfile                                                           0.0s
 => => transferring dockerfile: 32B                                                                                  0.0s
 => [pkg27 internal] load build definition from Dockerfile                                                           0.0s
 => => transferring dockerfile: 32B                                                                                  0.0s
 => [pkg29 internal] load build definition from Dockerfile                                                           0.0s
 => => transferring dockerfile: 32B                                                                                  0.0s
 => [pkg24 internal] load build definition from Dockerfile                                                           0.0s
 => => transferring dockerfile: 32B                                                                                  0.0s
 => [pkg09 internal] load build definition from Dockerfile                                                           0.0s
 => => transferring dockerfile: 32B                                                                                  0.0s
 => [pkg22 internal] load build definition from Dockerfile                                                           0.0s
 => => transferring dockerfile: 32B                                                                                  0.0s
 => [pkg16 internal] load build definition from Dockerfile                                                           0.0s
 => => transferring dockerfile: 32B                                                                                  0.0s
 => [pkg01 internal] load build definition from Dockerfile                                                           0.0s
 => => transferring dockerfile: 32B                                                                                  0.0s
 => [pkg07 internal] load build definition from Dockerfile                                                           0.0s
 => => transferring dockerfile: 32B                                                                                  0.0s
 => [pkg11 internal] load build definition from Dockerfile                                                           0.0s
 => => transferring dockerfile: 32B                                                                                  0.0s
 => [pkg31 internal] load build definition from Dockerfile                                                           0.0s
 => => transferring dockerfile: 32B                                                                                  0.0s
 => [pkg34 internal] load build definition from Dockerfile                                                           0.0s
 => => transferring dockerfile: 32B                                                                                  0.0s
 => [pkg17 internal] load build definition from Dockerfile                                                           0.0s
 => => transferring dockerfile: 32B                                                                                  0.0s
 => [pkg26 internal] load build definition from Dockerfile                                                           0.0s
 => => transferring dockerfile: 32B                                                                                  0.0s
 => [pkg04 internal] load .dockerignore                                                                              0.0s
 => => transferring context: 2B                                                                                      0.0s
 => [pkg12 internal] load .dockerignore                                                                              0.0s
 => => transferring context: 2B                                                                                      0.0s
 => [pkg32 internal] load .dockerignore                                                                              0.0s
 => => transferring context: 2B                                                                                      0.0s
 => [pkg30 internal] load .dockerignore                                                                              0.0s
 => => transferring context: 2B                                                                                      0.0s
 => [pkg25 internal] load .dockerignore                                                                              0.0s
 => => transferring context: 2B                                                                                      0.0s
 => [pkg05 internal] load .dockerignore                                                                              0.0s
 => => transferring context: 2B                                                                                      0.0s
 => [pkg18 internal] load .dockerignore                                                                              0.0s
 => => transferring context: 2B                                                                                      0.0s
 => [pkg14 internal] load .dockerignore                                                                              0.0s
 => => transferring context: 2B                                                                                      0.0s
 => [pkg15 internal] load .dockerignore                                                                              0.0s
 => => transferring context: 2B                                                                                      0.0s
 => [pkg02 internal] load .dockerignore                                                                              0.0s
 => => transferring context: 2B                                                                                      0.0s
 => [pkg10 internal] load .dockerignore                                                                              0.0s
 => => transferring context: 2B                                                                                      0.0s
 => [pkg38 internal] load .dockerignore                                                                              0.0s
 => => transferring context: 2B                                                                                      0.0s
 => [pkg36 internal] load .dockerignore                                                                              0.0s
 => => transferring context: 2B                                                                                      0.0s
 => [pkg19 internal] load .dockerignore                                                                              0.0s
 => => transferring context: 2B                                                                                      0.0s
 => [pkg23 internal] load .dockerignore                                                                              0.0s
 => => transferring context: 2B                                                                                      0.0s
 => [pkg39 internal] load .dockerignore                                                                              0.0s
 => => transferring context: 2B                                                                                      0.0s
 => [pkg03 internal] load .dockerignore                                                                              0.0s
 => => transferring context: 2B                                                                                      0.0s
 => [pkg28 internal] load .dockerignore                                                                              0.0s
 => => transferring context: 2B                                                                                      0.0s
 => [pkg08 internal] load .dockerignore                                                                              0.0s
 => => transferring context: 2B                                                                                      0.0s
 => [pkg37 internal] load .dockerignore                                                                              0.0s
 => => transferring context: 2B                                                                                      0.0s
 => [pkg21 internal] load .dockerignore                                                                              0.0s
 => => transferring context: 2B                                                                                      0.0s
 => [pkg35 internal] load .dockerignore                                                                              0.0s
 => => transferring context: 2B                                                                                      0.0s
 => [pkg20 internal] load .dockerignore                                                                              0.0s
 => => transferring context: 2B                                                                                      0.0s
 => [pkg06 internal] load .dockerignore                                                                              0.0s
 => => transferring context: 2B                                                                                      0.0s
 => [pkg00 internal] load .dockerignore                                                                              0.0s
 => => transferring context: 2B                                                                                      0.0s
 => [pkg13 internal] load .dockerignore                                                                              0.0s
 => => transferring context: 2B                                                                                      0.0s
 => [pkg33 internal] load .dockerignore                                                                              0.0s
 => => transferring context: 2B                                                                                      0.0s
 => [pkg27 internal] load .dockerignore                                                                              0.0s
 => => transferring context: 2B                                                                                      0.0s
 => [pkg29 internal] load .dockerignore                                                                              0.0s
 => => transferring context: 2B                                                                                      0.0s
 => [pkg24 internal] load .dockerignore                                                                              0.0s
 => => transferring context: 2B                                                                                      0.0s
 => [pkg09 internal] load .dockerignore                                                                              0.0s
 => => transferring context: 2B                                                                                      0.0s
 => [pkg22 internal] load .dockerignore                                                                              0.0s
 => => transferring context: 2B                                                                                      0.0s
 => [pkg16 internal] load .dockerignore                                                                              0.0s
 => => transferring context: 2B                                                                                      0.0s
 => [pkg01 internal] load .dockerignore                                                                              0.0s
 => => transferring context: 2B                                                                                      0.0s
 => [pkg07 internal] load .dockerignore                                                                              0.0s
 => => transferring context: 2B                                                                                      0.0s
 => [pkg11 internal] load .dockerignore                                                                              0.0s
 => => transferring context: 2B                                                                                      0.0s
 => [pkg31 internal] load .dockerignore                                                                              0.0s
 => => transferring context: 2B                                                                                      0.0s
 => [pkg34 internal] load .dockerignore                                                                              0.0s
 => => transferring context: 2B                                                                                      0.0s
 => [pkg17 internal] load .dockerignore                                                                              0.0s
 => => transferring context: 2B                                                                                      0.0s
 => [pkg26 internal] load .dockerignore                                                                              0.0s
 => => transferring context: 2B                                                                                      0.0s
 => [pkg26] resolve image config for docker.io/docker/dockerfile@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94  0.0s
 => CACHED [pkg26] docker-image://docker.io/docker/dockerfile@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e  0.0s
 => [pkg04 internal] load .dockerignore                                                                              0.0s
 => [pkg04 internal] load build definition from Dockerfile                                                           0.0s
 => [pkg26 internal] load metadata for docker.io/library/golang:1.19.2-alpine3.16@sha256:e4dcdac3ed37d8c2b3b8bcef29  0.0s
 => [pkg26 build 1/4] FROM docker.io/library/golang:1.19.2-alpine3.16@sha256:e4dcdac3ed37d8c2b3b8bcef2909573b2ad9c2  0.0s
 => CACHED [pkg26 build 2/4] WORKDIR /src                                                                            0.0s
 => CACHED [pkg26 build 3/4] COPY --from=base . ./                                                                   0.9s
 => CACHED [pkg26 build 4/4] RUN go build -o main ./pkg1/                                                            3.0s
 => [pkg12 internal] load build definition from Dockerfile                                                           0.0s
 => [pkg12 internal] load .dockerignore                                                                              0.0s
 => [pkg32 internal] load build definition from Dockerfile                                                           0.0s
 => [pkg32 internal] load .dockerignore                                                                              0.0s
 => [pkg30 internal] load build definition from Dockerfile                                                           0.0s
 => [pkg30 internal] load .dockerignore                                                                              0.0s
 => [pkg25 internal] load .dockerignore                                                                              0.0s
 => [pkg25 internal] load build definition from Dockerfile                                                           0.0s
 => [pkg05 internal] load build definition from Dockerfile                                                           0.0s
 => [pkg05 internal] load .dockerignore                                                                              0.0s
 => [pkg18 internal] load .dockerignore                                                                              0.0s
 => [pkg18 internal] load build definition from Dockerfile                                                           0.0s
 => [pkg14 internal] load build definition from Dockerfile                                                           0.0s
 => [pkg14 internal] load .dockerignore                                                                              0.0s
 => [pkg15 internal] load build definition from Dockerfile                                                           0.0s
 => [pkg15 internal] load .dockerignore                                                                              0.0s
 => [pkg02 internal] load build definition from Dockerfile                                                           0.0s
 => [pkg02 internal] load .dockerignore                                                                              0.0s
 => [pkg10 internal] load .dockerignore                                                                              0.0s
 => [pkg10 internal] load build definition from Dockerfile                                                           0.0s
 => [pkg38 internal] load build definition from Dockerfile                                                           0.0s
 => [pkg38 internal] load .dockerignore                                                                              0.0s
 => [pkg36 internal] load .dockerignore                                                                              0.0s
 => [pkg36 internal] load build definition from Dockerfile                                                           0.0s
 => [pkg19 internal] load .dockerignore                                                                              0.0s
 => [pkg19 internal] load build definition from Dockerfile                                                           0.0s
 => [pkg23 internal] load build definition from Dockerfile                                                           0.0s
 => [pkg23 internal] load .dockerignore                                                                              0.0s
 => [pkg39 internal] load .dockerignore                                                                              0.0s
 => [pkg39 internal] load build definition from Dockerfile                                                           0.0s
 => [pkg03 internal] load .dockerignore                                                                              0.0s
 => [pkg03 internal] load build definition from Dockerfile                                                           0.0s
 => [pkg28 internal] load build definition from Dockerfile                                                           0.0s
 => [pkg28 internal] load .dockerignore                                                                              0.0s
 => [pkg08 internal] load build definition from Dockerfile                                                           0.0s
 => [pkg08 internal] load .dockerignore                                                                              0.0s
 => [pkg37 internal] load .dockerignore                                                                              0.0s
 => [pkg37 internal] load build definition from Dockerfile                                                           0.0s
 => [pkg21 internal] load build definition from Dockerfile                                                           0.0s
 => [pkg21 internal] load .dockerignore                                                                              0.0s
 => [pkg35 internal] load build definition from Dockerfile                                                           0.0s
 => [pkg35 internal] load .dockerignore                                                                              0.0s
 => [pkg20 internal] load build definition from Dockerfile                                                           0.0s
 => [pkg20 internal] load .dockerignore                                                                              0.0s
 => [pkg06 internal] load build definition from Dockerfile                                                           0.0s
 => [pkg06 internal] load .dockerignore                                                                              0.0s
 => [pkg00 internal] load build definition from Dockerfile                                                           0.0s
 => [pkg00 internal] load .dockerignore                                                                              0.0s
 => [pkg13 internal] load build definition from Dockerfile                                                           0.0s
 => [pkg13 internal] load .dockerignore                                                                              0.0s
 => CACHED [pkg26 stage-1 1/1] COPY --from=build /src/main /main                                                     0.0s
 => [pkg33 internal] load .dockerignore                                                                              0.0s
 => [pkg33 internal] load build definition from Dockerfile                                                           0.0s
 => [pkg29 internal] load build definition from Dockerfile                                                           0.0s
 => [pkg29 internal] load .dockerignore                                                                              0.0s
 => [pkg27 internal] load build definition from Dockerfile                                                           0.0s
 => [pkg27 internal] load .dockerignore                                                                              0.0s
 => [pkg24 internal] load .dockerignore                                                                              0.0s
 => [pkg24 internal] load build definition from Dockerfile                                                           0.0s
 => [pkg09 internal] load .dockerignore                                                                              0.0s
 => [pkg09 internal] load build definition from Dockerfile                                                           0.0s
 => [pkg22 internal] load .dockerignore                                                                              0.0s
 => [pkg22 internal] load build definition from Dockerfile                                                           0.0s
 => [pkg16 internal] load .dockerignore                                                                              0.0s
 => [pkg16 internal] load build definition from Dockerfile                                                           0.0s
 => [pkg01 internal] load .dockerignore                                                                              0.0s
 => [pkg01 internal] load build definition from Dockerfile                                                           0.0s
 => [pkg07 internal] load .dockerignore                                                                              0.0s
 => [pkg07 internal] load build definition from Dockerfile                                                           0.0s
 => [pkg11 internal] load build definition from Dockerfile                                                           0.0s
 => [pkg11 internal] load .dockerignore                                                                              0.0s
 => [pkg31 internal] load .dockerignore                                                                              0.0s
 => [pkg31 internal] load build definition from Dockerfile                                                           0.0s
 => [pkg34 internal] load build definition from Dockerfile                                                           0.0s
 => [pkg34 internal] load .dockerignore                                                                              0.0s
 => [pkg17 internal] load build definition from Dockerfile                                                           0.0s
 => [pkg17 internal] load .dockerignore                                                                              0.0s
 => [pkg26 internal] load .dockerignore                                                                              0.0s
 => [pkg26 internal] load build definition from Dockerfile                                                           0.0s

Versions used:

❯ docker version
Client:
 Cloud integration: v1.0.29
 Version:           20.10.20
 API version:       1.41
 Go version:        go1.18.7
 Git commit:        9fdeb9c
 Built:             Tue Oct 18 18:20:35 2022
 OS/Arch:           darwin/arm64
 Context:           default
 Experimental:      true

Server: Docker Desktop 4.13.0 (89412)
 Engine:
  Version:          20.10.20
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.7
  Git commit:       03df974
  Built:            Tue Oct 18 18:18:16 2022
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.8
  GitCommit:        9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

❯ docker buildx version
github.com/docker/buildx v0.9.1 ed00243a0ce2a0aee75311b06e32d33b44729689

❯ docker buildx ls
NAME/NODE       DRIVER/ENDPOINT STATUS  BUILDKIT PLATFORMS
default *       docker
  default       default         running 20.10.20 linux/arm64, linux/amd64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6
desktop-linux   docker
  desktop-linux desktop-linux   running 20.10.20 linux/arm64, linux/amd64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

[tonistiigi]

Same as #92

You can see the previous work linked there. This is possible in buildkit side, but maybe could use reevaluation of the design. You can make builds with shared sessions but I think the cases that are not possible atm would be when two builds share the same build context but want to pass different secrets(as both as sent via session). I know @aaronlehmann uses the shared session implementation so maybe he has some ideas here.

With the current buildkit implementation, if we detect the chases where parallel builds are sharing the session fully(we need to check that they all use the same build secrets/ssh config, as well as the same build context of course) we could make them use the shared session.

[dgageot]

Thank you @tonistiigi for the detailed feedback!

Of course that would be awesome if bake could sort out by itself that sessions can be shared safely.
On my side, I don't mind giving a hint to bake. This is basically what I do by creating an intermediate base target that copies from the context and from which the bake target can copy. That gives me better perf already. However, I'm wondering if I could do that without changing the Dockerfiles and only on the bake side.

First, I thought I could "just" add this base target to the bake file and use the contexts directive to make . point to target:base instead. That doesn't seem possible. I'm wondering if it could be an easy patch. To me, that seems like a useful thing to allow. wdyt?

[tonistiigi]

First, I thought I could "just" add this base target to the bake file and use the contexts directive to make . point to target:base instead.

This should be possible on the buildkit side if buildx would pass the correct argument (and define how user describes this case).

The problem with this is though that context can be selectively transferred based on what files the dockerfile stages use. Maybe it makes more sense to just use a base stage in a Dockerfile that figures what files are needed.

Smth. like:

from alpine as base
copy . /ctx

from base as target1
run apk add git
run ls -l /ctx

from base as target2
run apk add curl
run ls -l /ctx

group "default" {
  targets = ["target1", "target2"]
}

target "base" {
  target = "base"
}

target "target1" {
  target = "target1"
  contexts = {
    base = "target:base"
  }
}

target "target2" {
  target = "target2"
  contexts = {
    base = "target:base"
  }
}

If just using context makes more sense

from alpine as base

from scratch as ctx
copy . .

from base as target1
run apk add git
run --mount=target=ctx,from=ctx ls -l

from base as target2
run apk add curl
run --mount=target=ctx ls -l /ctx

group "default" {
  targets = ["target1", "target2"]
}

target "ctx" {
  target = "ctx"
}

target "target1" {
  target = "target1"
  contexts = {
    ctx = "target:ctx"
  }
}

target "target2" {
  target = "target2"
  contexts = {
    ctx = "target:ctx"
  }
}

[tonistiigi]

I tried the last example with

from scratch as ctx
copy --link . .

and looks like it is smart enough to completely optimize out the copy step in that case and both targets just use the context directly.

[bhavitsharma]

+1 for fixing this issue; this is a massive issue for us. Context transfer for 15 parallel builds of our images takes around ~200 seconds which is insane. Our dockerfile build is slower than our entire bazel build of the project, and all our dockerfiles are just copying the build artefacts. We reduced parallelism to reduce build time, but the context transfer still takes much time. I would love to see this issue resolved!

We're thinking of completely switching to podman with volume mounts and --no-cache because that's much faster than using buildkit.