failed to solve (mounting not permitted) when using namespace remapping

[joshspicer]

I am attempting to utilize docker namespace remapping. Builds are unsuccessful with Dockerfiles that include a syntax directive (Eg # syntax=docker/dockerfile:1.4) with ERROR: failed to solve: exit code: 1.

An example Dockerfile:

#syntax=docker/dockerfile:1.4
FROM mcr.microsoft.com/devcontainers/base

# docker build -t test .
[+] Building 1.1s (5/5) FINISHED                                                                                                                                                         docker:default
 => [internal] load build definition from Dockerfile                                                                                                                                               0.1s
 => => transferring dockerfile: 110B                                                                                                                                                               0.0s
 => [internal] load .dockerignore                                                                                                                                                                  0.1s
 => => transferring context: 2B                                                                                                                                                                    0.0s
 => resolve image config for docker.io/docker/dockerfile:1.4                                                                                                                                       0.7s
 => [auth] docker/dockerfile:pull token for registry-1.docker.io                                                                                                                                   0.0s
 => CACHED docker-image://docker.io/docker/dockerfile:1.4@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc                                                                  0.0s
Dockerfile:1
--------------------
   1 | >>> # syntax=docker/dockerfile:1.4
   2 |     FROM mcr.microsoft.com/devcontainers/base
   3 |     
--------------------
ERROR: failed to solve: exit code: 1

The following 'operation not permitted' error can be found in the journal logs:

Jan 13 00:40:10 f5e30636-1287-4fa2-82c3-c4fee7233074 dockerd[16476]: 

runc run failed: unable to start container process: error during container init: error mounting 
"/var/lib/docker/231072.231072/tmp/buildkit-metadata1667912507" to rootfs at 
"/run/config/buildkit/metadata": mount /proc/self/fd/8:/run/config/buildkit/metadata (via 
/proc/self/fd/9), flags: 0x1021: operation not permitted

This repros on a default Ubuntu 22.04 Azure VM, with some limited customizations. One notable customization here is that the /var/lib/docker directory is a mount of an ext4 filesystem (more details hidden below). I don't believe this is the cause of the issue, but thought i'd point it out for added context.

I have no issues building docker images on this host unless combining with user namespace remapping. What i've tried:

✅ A Dockerfile without the syntax directive build and WITH user namespace remapping
✅ Building Dockerfile WITH syntax directive and WITHOUT user namespace remapping
 

docker info

 $ /bin/sh -c docker info && uname -a
 Client:
 Version:    24.0.7-1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  0.12.0-1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
2024-01-13 00:52:12.136Z:   compose: Docker Compose (Docker Inc.)
    Version:  2.23.3-1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 1
 Server Version: 24.0.7-1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: ********
  Using metacopy: false
  Native Overlay Diff: false
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3dd1e886e55dd695541fdcd67420c2888645a495
 runc version: ccaecfcbc907d70a7aa870a6650887b901b25b82
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  userns
  cgroupns
 Kernel Version: 6.2.0-1018-azure
 Operating System: Ubuntu 22.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 3.812GiB
 Name: 8c3ef404-b5db-4e2a-ac6b-b36b51343d9b
 ID: 0c258cbb-762f-4ba7-972b-6dedb9d9ce83
 Docker Root Dir: /var/lib/docker/231072.231072
 Debug Mode: false
 Username: codespacesdev
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

/etc/docker/daemon.json

{
	"userns-remap": "default"
}

mount

/dev/sdb1 on / type ext4 (rw,relatime,discard,errors=remount-ro)
devtmpfs on /dev type devtmpfs (rw,nosuid,noexec,relatime,size=1996516k,nr_inodes=499129,mode=755,inode64)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,size=799536k,nr_inodes=819200,mode=755,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13938)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
ramfs on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
/var/lib/snapd/snaps/core20_2015.snap on /snap/core20/2015 type squashfs (ro,nodev,relatime,errors=continue,threads=single,x-gdu.hide)
/var/lib/snapd/snaps/lxd_24322.snap on /snap/lxd/24322 type squashfs (ro,nodev,relatime,errors=continue,threads=single,x-gdu.hide)
/var/lib/snapd/snaps/snapd_20290.snap on /snap/snapd/20290 type squashfs (ro,nodev,relatime,errors=continue,threads=single,x-gdu.hide)
/dev/sdb15 on /boot/efi type vfat (rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
/dev/sda1 on /mnt type ext4 (rw,relatime,x-systemd.requires=cloud-init.service,_netdev)
tmpfs on /run/snapd/ns type tmpfs (rw,nosuid,nodev,size=799536k,nr_inodes=819200,mode=755,inode64)
nsfs on /run/snapd/ns/lxd.mnt type nsfs (rw)
/var/lib/snapd/snaps/snapd_20671.snap on /snap/snapd/20671 type squashfs (ro,nodev,relatime,errors=continue,threads=single,x-gdu.hide)
/var/lib/snapd/snaps/core20_2105.snap on /snap/core20/2105 type squashfs (ro,nodev,relatime,errors=continue,threads=single,x-gdu.hide)
//vsocef459ab970a3b4b28b5f.z21.file.storage.azure.net/cloudenvdata on /mnt/cloudenvdata type cifs (rw,relatime,vers=3.1.1,cache=strict,username=vsocef459ab970a3b4b28b5f,uid=0,noforceuid,gid=0,noforcegid,addr=52.239.237.8,file_mode=0777,dir_mode=0777,soft,persistenthandles,nounix,serverino,mapposix,rsize=1048576,wsize=1048576,bsize=1048576,echo_interval=60,actimeo=1,closetimeo=1)
/dev/loop5 on /var/lib/docker type ext4 (rw,nodev,relatime)
/dev/loop6 on /var/log/codespaces/backup type ext4 (rw,nodev,relatime)

OS/kernel info

# cat /etc/os-release 
PRETTY_NAME="Ubuntu 22.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

# uname -a
Linux 6e4d4f2d-4af8-4041-95b3-06a832ca6b35 6.2.0-1018-azure #18~22.04.1-Ubuntu SMP Tue Nov 21 19:25:02 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Repro in codespaces

Edit: This also repros easily in a codespace with the following steps:

1. Create a GitHub Codespace on nearly any repo (can't have a custom devcontainer.json unless it also includes docker. I've tested with the official docker-in-docker v2.7.1 Feature).
   1. Using this repo moby/buildkit works fine for the repro.
2. Enable user namespace remapping in /etc/docker/daemon.json.
3. Kill and restart dockerd (pkill dockerd && dockerd).
4. Try to build the Dockerfile above

docker info

@joshspicer ➜ /workspaces/buildkit (master) $ docker info && uname -a
Client:
 Version:    24.0.7-1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  0.12.0-1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  2.23.3-1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 24.0.7-1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: false
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3dd1e886e55dd695541fdcd67420c2888645a495
 runc version: 4bccb38cc9cf198d52bebf2b3a90cd14e7af8c06
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  userns
  cgroupns
 Kernel Version: 6.2.0-1018-azure
 Operating System: Ubuntu 20.04.6 LTS (containerized)
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 7.744GiB
 Name: codespaces-7372cd
 ID: 65629d15-48c3-4cf1-adef-23a3c0da7b2a
 Docker Root Dir: /var/lib/docker/165536.165536
 Debug Mode: false
 Username: codespacesdev
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Linux codespaces-7372cd 6.2.0-1018-azure #18~22.04.1-Ubuntu SMP Tue Nov 21 19:25:02 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

[joshspicer]

The only similar discussion i've found online thus far is containerd/nerdctl#1401 .

There are a non-zero number people in our userbase that seem to rely on providing the syntax directive (microsoft/vscode-remote-release#7463) so just stripping it from Dockerfiles doesn't seem like an option (and is probably hiding a deeper issue)

[joshspicer]

Possibly similar error found in the rootless docs: https://github.com/moby/buildkit/blob/master/docs/rootless.md#error-mount-procproc-via-procselffd6-flags-0xe-operation-not-permitted

[joshspicer]

I do notice that the tmp directory under the remapped user's /var/lib/docker is not in the same group as some of the other directories. I've tried creating this and manually changing ownership and updating permissions with the sticky bit (g+rwxs), but it didn't seem to help in this case

[joshspicer]

@tonistiigi Do you perhaps have any quick suggestions I could try to get around this issue? 🙏

[tonistiigi]

It seems to be an issue with this mount https://github.com/moby/buildkit/blob/master/frontend/gateway/gateway.go#L341-L348 not converting the path to file accessible for the remapped user. worker.CacheManger().IdentityMapping() should give the parameters needed fix the permissions.

[joshspicer]

FWIW i've found a much easier way to repro in Codespaces. I've edited the issue description above (see the Repro in codespaces section)

[joshspicer]

I think some solution similar to #3098 may help here.

Any tips on running/simulating a development builder under a user namespace? I see the buildkitd --rootless flag but i'm not yet able to repro this issue compiling from source. I'd love to get a repro building from source so I could test a potential fix (cc/ @tonistiigi if you have a doc or quick suggestion here). Is running buildkitd in a linux username enough to simulate the user-ns docker flag?

[tonistiigi]

I could not reproduce this in v25 of moby

[tonistiigi]

This indeed produces in codespaces but not in any other system I've tested. Also does not produce in dind. I thought that this was maybe apparmor, but even in systems with apparmor enabled I see no issue. As codespaces does not use official Docker release (and I don't know how the VM is set up) can't debug this more before additional info is provided.

[joshspicer]

Thank you for resolving 🙏