Inline caching breaks docker history when using WORKDIR

[zachary-povey]

Hi,

I am finding that when using inline caching with buildkit, images built with this cache display incorrect layer sizes when inspected using docker history. The problem seems to occur only for Dockerfiles where WORKDIR is used (at least this is the only cause I have so far seen).

Steps to reproduce

To reproduce this issue consistently it is really important to ensure your builder has an empty cache, I have made a little repo with some supporting scripts to help reproduce the issue consistently here, I really suggest you use this to help you reproduce. But key points are, using this Dockerfile:

FROM rockylinux/rockylinux:8

# Create an empty dir
RUN mkdir /some_dir

# Set working dir - this is the line which seems to get stuff mixed up - without it, the layer sizes make sense
WORKDIR /some_dir

# Download a 10kB file
RUN curl -X GET "https://httpbin.org/bytes/10000" -H "accept: application/octet-stream" > ./dump

If you build the image with a buildx builder (buildkit >= 0.8.0) then run docker history you get this:

IMAGE          CREATED         CREATED BY                                      SIZE      COMMENT
400e454bd6db   5 seconds ago   RUN /bin/sh -c curl -X GET "https://httpbin.…   10kB      buildkit.dockerfile.v0
<missing>      5 seconds ago   WORKDIR /some_dir                               0B        buildkit.dockerfile.v0
<missing>      5 seconds ago   RUN /bin/sh -c mkdir /some_dir # buildkit       0B        buildkit.dockerfile.v0
<missing>      7 weeks ago     CMD ["/bin/bash"]                               0B        buildkit.dockerfile.v0
<missing>      7 weeks ago     LABEL org.label-schema.schema-version=1.0 or…   0B        buildkit.dockerfile.v0
<missing>      7 weeks ago     ADD rocky-8.5-docker-aarch64.tar.xz / # buil…   241MB     buildkit.dockerfile.v0

Here the layers make sense for the given commands: note how the last line downloads a 10kB file from httpbin and as we might expect this results in a 10kB layer.

If you:

• build this image with buildx, docker container driver (buildkit version >= 0.8.0) and set --cache-to=type=inline
• push it to a registry (as <registry>/<image>:<tag>)
• build again with a fresh builder (no local cache) and --cache-from=<registry>/<image>:<tag>
• re-run docker history we get:

IMAGE          CREATED          CREATED BY                                      SIZE      COMMENT
108c1f87a532   19 seconds ago   RUN /bin/sh -c curl -X GET "https://httpbin.…   0B        buildkit.dockerfile.v0
<missing>      19 seconds ago   WORKDIR /some_dir                               10kB      buildkit.dockerfile.v0
<missing>      19 seconds ago   RUN /bin/sh -c mkdir /some_dir # buildkit       0B        buildkit.dockerfile.v0
<missing>      7 weeks ago      CMD ["/bin/bash"]                               0B        buildkit.dockerfile.v0
<missing>      7 weeks ago      LABEL org.label-schema.schema-version=1.0 or…   0B        buildkit.dockerfile.v0
<missing>      7 weeks ago      ADD rocky-8.5-docker-aarch64.tar.xz / # buil…   241MB     buildkit.dockerfile.v0

Notice how it now suggests the WORKDIR line created a 10kB layer and the download of a 10kB file is an empty layer which seems completely wrong.

So far I have only noticed this for Dockerfiles using WORKDIR. I have reproduced the issue for buildkit v0.8.0-0.9.3 - using v0.7.0 gave history as expected (the same both times).

Confusingly, I was unable to reproduce the issue using the default buildx builder even though based on my docker engine version (20.10.11) I expected to be using buildkit version 0.8.3 based on this.

Again I strongly suggest using my helper scripts to reproduce the issue.

Local env

docker version

Client:
 Cloud integration: v1.0.22
 Version:           20.10.11
 API version:       1.41
 Go version:        go1.16.10
 Git commit:        dea9396
 Built:             Thu Nov 18 00:36:09 2021
 OS/Arch:           darwin/arm64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.11
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.9
  Git commit:       847da18
  Built:            Thu Nov 18 00:34:44 2021
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.4.12
  GitCommit:        7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker buildx version

github.com/docker/buildx v0.7.1 05846896d149da05f3d6fd1e7770da187b52a247

[tonistiigi]

This is indeed a known issue described in some other inline-cache issues/pr-s.

Only solution I currently have is to remove the "empty layer detection" code from buildkit and start pushing blobs with no contents. The issue should be completely visual in the history array though. No dropped layers or changes in files etc.

[mbentley]

I will say that this threw me for a bit of a loop - I have been running some regular checks to look at the layer sizes as a result of the zero byte layers that were showing up from #2198 and I was wondering why I was seeing false positives.

[tonistiigi]

Should be fixed with #2651