Image built with missing layers (size = 0B) !!

[carlosgalvezp]

Hi,

We recently switched over to buildkit for building our docker images (via DOCKER_BUILDKIT=1) to take advantage of the latest features, such as secrets.
Our Dockerfile performs a multi-stage build with N temporary images. In the final image, we copy contents (built binaries) from these N temporary images into the final image.

We build our images in a CI fashion. We also make use of cache via:
--build-arg=BUILDKIT_INLINE_CACHE=1
--cache-from=registry:port/image:latest

So the workflow is:
docker pull registry:port/image:latest
docker build registry:port/image:latest
docker push registry:port/image:latest

This worked fine the first time. However the second time building the same image, we notice that it ends up with missing content!! Layers that should occupy a few MB have a size of 0B:

Example:
IMAGE CREATED CREATED BY SIZE COMMENT f581b8108b78 26 hours ago COPY /tmp/install / # buildkit 0B buildkit.dockerfile.v0 <missing> 26 hours ago COPY /tmp/install / # buildkit 0B buildkit.dockerfile.v0 <missing> 26 hours ago COPY /tmp/install / # buildkit 0B buildkit.dockerfile.v0 <missing> 26 hours ago COPY /tmp/install / # buildkit 0B buildkit.dockerfile.v0 <missing> 26 hours ago COPY /tmp/install / # buildkit 0B buildkit.dockerfile.v0 <missing> 26 hours ago COPY /tmp/install / # buildkit 0B buildkit.dockerfile.v0 <missing> 26 hours ago COPY /tmp/install / # buildkit 0B buildkit.dockerfile.v0 <missing> 26 hours ago COPY /tmp/install / # buildkit 0B buildkit.dockerfile.v0 <missing> 26 hours ago COPY /tmp/install / # buildkit 0B buildkit.dockerfile.v0 <missing> 26 hours ago COPY /tmp/install / # buildkit 0B buildkit.dockerfile.v0 <missing> 26 hours ago COPY /tmp/install / # buildkit 0B buildkit.dockerfile.v0 <missing> 26 hours ago COPY /tmp/install / # buildkit 3.06MB buildkit.dockerfile.v0 <missing> 26 hours ago COPY /tmp/install / # buildkit 1.14MB buildkit.dockerfile.v0 <missing> 26 hours ago COPY /tmp/install / # buildkit 3.49MB buildkit.dockerfile.v0
You can see the COPY calls. Some of them do actually copy (non-zero size) but most of them don't.

We noticed that the intermediate images for which COPY works had to be rebuilt again - i.e. were not found in the cache (even though the contents of the Dockerfile for that part didn't change at all). Intermediate images that did hit the cache (CACHED) are the ones that show this problem, it's like they are empty.

Why is this happening? We are using docker 19.03

[scaytrase]

If you are using secrets, maybe you are facing the same issue we did. Something is broken in master-experimental. Latest build of new features which runs OK for us is

#syntax=docker/dockerfile-upstream:20191217-secrets

All other gives us things like

unable to find user www-data: no matching entries in passwd file

which seems related to broken images, since event running simple RUN ls -lsa gives us an error on inherited image

[tonistiigi]

@scaytrase If you have an issue with secrets on the master branch please create a new issue with a reproducer.

[scaytrase]

I managed to create reproducing case, will report standalone issue

[scaytrase]

#1327