Mounted dirs get added to layer changeset, is this expected?

[aconz2]

I was a bit surprised to find that the layers from a built container have the dirs that are mounted in by the container runtime like proc run sys and sometimes etc.

This script demonstrates the behavior, both with and without --dns=none --no-hosts --no-hostname to show etc (I originally thought this wasn't working to exclude etc but podman build will reuse the build cache even when it gets different arguments so I was thrown off at first)

set -e

function get_layers() {
    image_manifest=$(jq -r '.manifests[0].digest' index.json | sed 's_:_/_')
    jq -r '.layers[].digest' blobs/$image_manifest | sed 's_:_/_'
}

function show_layers() {
    pushd . &>/dev/null
    cd "$1"
    for layer in $(get_layers | tail -n+2); do
        echo $layer
        tar tvf blobs/$layer
        echo
    done
    popd &>/dev/null
}

mkdir -p /tmp/question
cd /tmp/question
name=githubquestion

# use Dockerfile name b/c I don't know how to get buildkit to use a different name
cat << "EOF" > Dockerfile
FROM docker.io/library/busybox
RUN echo hi > a
RUN echo hi > b
EOF

trap 'trap - SIGTERM && kill 0' SIGINT SIGTERM EXIT

# NOTE: we have to remove the image between builds otherwise it will use the build cache
# even though we use --dns=none --no-hosts --no-hostname the second time around

echo "# podman build -f Dockerfile"
podman rmi --ignore $name >/dev/null
podman build -f Dockerfile -t $name >/dev/null
rm -rf oci-dir && mkdir oci-dir
podman save --format oci-archive $name | tar xf - -C oci-dir
show_layers oci-dir

echo -e "---------------------------------\n"

echo "# podman build --dns=none --no-hosts --no-hostname -f Dockerfile"
podman rmi $name >/dev/null
podman build --dns=none --no-hosts --no-hostname -f Dockerfile -t $name >/dev/null
rm -rf oci-dir && mkdir oci-dir
podman save --format oci-archive $name | tar xf - -C oci-dir
show_layers oci-dir

echo -e "---------------------------------\n"

mkdir -p varrun
trap 'kill $(jobs -p)' EXIT

echo "# docker build . (containerized docker)"
podman run --privileged --rm -v $(realpath varrun):/var/run/ docker:latest &>/dev/null &
sleep 2  # wait for daemon to load
podman run --privileged --rm -v $(realpath varrun):/var/run -v $(realpath .):/$(realpath .) -w $(realpath .) docker:latest build -t $name . &>/dev/null
rm -rf oci-dir && mkdir oci-dir
podman run --privileged --rm -v $(realpath varrun):/var/run -v $(realpath .):/$(realpath .) -w $(realpath .) docker:latest save $name | tar xf - -C oci-dir
show_layers oci-dir
kill %%

echo -e "---------------------------------\n"

# varrun (from above) has root owned files, docker fails when trying to load them as context
mkdir -p clean
cp Dockerfile clean/
cd clean

# NOTE: for this I have an externally running command like
# wget https://download.docker.com/linux/static/stable/x86_64/docker-27.3.1.tgz
# tar xf docker-27.3.1.tgz  # unpacks a docker dir
# wget https://github.com/moby/buildkit/releases/download/v0.17.0/buildkit-v0.17.0.linux-amd64.tar.gz
# tar xf buildkit-v0.17.0.linux-amd64.tar.gz  # unpacks a bin dir
# PATH=$(realpath docker):$PATH sudo --preserve-env=PATH dockerd
# PATH=$(realpath bin):$PATH    sudo --preserve-env=PATH buildkitd
# sudo chown $USER:$USER /var/run/docker.sock
# sudo chown $USER:$USER /var/run/buildkit/buildkitd.sock

echo "# docker build . (non-containerized docker)"
# tried to get this to work but to no avail
#sudo --preserve-env=PATH dockerd &
#sudo chown $USER:$USER /var/run/docker.sock
docker build -f Dockerfile -t $name . &>/dev/null
rm -rf oci-dir && mkdir oci-dir
docker save $name | tar xf - -C oci-dir
show_layers oci-dir

echo -e "---------------------------------\n"

echo "# buildctl build --frontend dockerfile.v0 --local dockerfile=. (non-containerized buildkit)"
# podman run --privileged --rm docker.io/moby/buildkit:latest &  # tried getting this to work but no avail
rm -rf oci-dir && mkdir oci-dir
#buildctl --addr=podman-container://buildkitd build --frontend dockerfile.v0 --local context=. --local dockerfile=. --output type=oci | tar xf - -C oci-dir
buildctl build --frontend dockerfile.v0 --local dockerfile=. --output type=oci 2>/dev/null | tar xf - -C oci-dir
show_layers oci-dir

# podman build -f Dockerfile
sha256/5cf159b25150548408f2ab7878ebfcbb72c7f4eb80e601b383c73a07466b563f
-rw-r--r-- 0/0               3 2024-11-04 12:53 a
drwxr-xr-x 0/0               0 2024-11-04 12:53 etc/
-rwx------ 0/0               0 2024-11-04 12:53 etc/hostname
-rwx------ 0/0               0 2024-11-04 12:53 etc/hosts
-rwx------ 0/0               0 2024-11-04 12:53 etc/resolv.conf
drwxr-xr-x 0/0               0 2024-11-04 12:53 proc/
drwxr-xr-x 0/0               0 2024-11-04 12:53 run/
drwxr-xr-x 0/0               0 2024-11-04 12:53 sys/

sha256/8084e52cf8f5f5226b22f51ea4bd83132881f956b1249fe8fea89899f2817117
-rw-r--r-- 0/0               3 2024-11-04 12:53 b
drwxr-xr-x 0/0               0 2024-11-04 12:53 run/

---------------------------------

# podman build --dns=none --no-hosts --no-hostname -f Dockerfile
sha256/ab00a8832731544da7f312ec1e9f01ff22ca1afcabe3e2588ba2f3a2c3b77f14
-rw-r--r-- 0/0               3 2024-11-04 12:53 a
drwxr-xr-x 0/0               0 2024-11-04 12:53 proc/
drwxr-xr-x 0/0               0 2024-11-04 12:53 run/
drwxr-xr-x 0/0               0 2024-11-04 12:53 sys/

sha256/36a6684b7dacab62cf8089aacd4f3c8a5888923285d53fed65636ac0a8026d66
-rw-r--r-- 0/0               3 2024-11-04 12:53 b
drwxr-xr-x 0/0               0 2024-11-04 12:53 run/

---------------------------------

# docker build . (containerized docker)
sha256/7d2e4233a7ca70daa5df8ff388d75998346b14773876ac4f921fd4a3812ec5ca
-rw-r--r-- 0/0               3 2024-11-04 12:53 a
drwxr-xr-x 0/0               0 2024-11-04 12:53 etc/
drwxr-xr-x 0/0               0 2024-11-04 12:53 proc/
drwxr-xr-x 0/0               0 2024-11-04 12:53 sys/

sha256/67ddece4132c33925e0913a0b719ef59729019f0498449869bcac20a95f4b37a
-rw-r--r-- 0/0               3 2024-11-04 12:53 b
drwxr-xr-x 0/0               0 2024-11-04 12:53 etc/

---------------------------------

# docker build . (non-containerized docker)
sha256/dd13708efb915f81d29acd524ac1968c8e7e3967150f545a6a95e0cfc4f232cf
-rw-r--r-- 0/0               3 2024-11-04 12:36 a

sha256/445b685946881ce0122e888d79d9a35a924a153d7c15051146660e055b234222
-rw-r--r-- 0/0               3 2024-11-04 12:36 b

---------------------------------

# buildctl build --frontend dockerfile.v0 --local dockerfile=. (non-containerized buildkit)
sha256/323c79c0f96ecc571434c734230394ed07c9aa7d93369859fd9183d2fb631fa4
-rw-r--r-- 0/0               3 2024-11-04 11:34 a
drwxr-xr-x 0/0               0 2024-11-04 11:34 proc/
drwxr-xr-x 0/0               0 2024-11-04 11:34 sys/

sha256/82ad12d453af67cc78efc6b9ce3f89806dd1e8134cb563dedaef3f0ffcf5432c
-rw-r--r-- 0/0               3 2024-11-04 11:34 b

The base layer (0) is not shown.

The first layer has etc/{hostname,hosts,resolv.conf} proc run sys and the remainder of the layers have a run dir (this holds for more layers in my testing). My assumption going in was that they would only contain the files/dirs that the RUN statement touches, and not anything from the container runtime. Interesting to note that hostname et al are 0 length.

I presume this is from those dirs being mounted in (and I guess more specifically those dirs have to be created in the workdir of the overlay to then get mounted in; not sure though why only run appears in all of them but not the other mount dirs)

Overall I'm just wondering if this is known/expected behavior and/or if other container runtimes/builders exhibit the same behavior (I will try to do some testing when I get a chance). And does the OCI spec have anything to say about this? I'm coming at this from a reproducibility standpoint so having these container runtime artifacts appear is not ideal.

UPDATE: I tested with both docker in container, docker out of container, and buildkit and get different results from all of them.

[Luap99]

Note this is tracked as issue in containers/buildah#4242

[aconz2]

Ahh interesting, thank you!