Podman Error: mount proc to /proc: Operation not permitted: OCI permission denied (Chromeos/Crostini/Debian 10)

[ctophs]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Podman does not seem to be able to start a container.

tl;dr:

podman run --rm 2f4357dd9647 /bin/echo "fubar"
Error: mount `proc` to `/proc`: Operation not permitted: OCI permission denied

Steps to reproduce the issue:

Long Version:

According to https://podman.io/getting-started/installation

this is only needed on real debian kernels while crostini uses its own so I skipped that.

# Debian 10
# First enable user namespaces as root user
echo 'kernel.unprivileged_userns_clone=1' > /etc/sysctl.d/00-local-userns.conf
systemctl restart procps

uname -a is

Linux penguin 5.4.88-12224-gf05236dbdecf #1 SMP PREEMPT Fri Feb 19 05:44:55 PST 2021 x86_64 GNU/Linux

going on with

# Use buster-backports on Debian 10 for a newer libseccomp2
echo 'deb http://deb.debian.org/debian buster-backports main' >> /etc/apt/sources.list
echo 'deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/Debian_10/ /' > /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list
curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/Debian_10/Release.key | sudo apt-key add -
sudo apt-get update
sudo apt-get -y -t buster-backports install libseccomp2

Since I wanted to go rootless I went on installing this

sudo apt-get -y install podman-rootless

results in..

sudo apt install podman-rootless
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  catatonit conmon containers-common criu crun dns-root-data dnsmasq-base libgpgme11 libnet1 libnl-3-200 libprotobuf17 libyajl2 podman-plugins python3-pkg-resources python3-protobuf
  python3-six slirp4netns uidmap
Suggested packages:
  python3-setuptools
The following NEW packages will be installed:
  catatonit conmon containers-common criu crun dns-root-data dnsmasq-base libgpgme11 libnet1 libnl-3-200 libprotobuf17 libyajl2 podman-plugins podman-rootless python3-pkg-resources
  python3-protobuf python3-six slirp4netns uidmap
0 upgraded, 19 newly installed, 0 to remove and 0 not upgraded.
Need to get 26.6 MB of archives.
After this operation, 114 MB of additional disk space will be used.
Do you want to continue? [Y/n] Y

installed.

# Restart dbus for rootless podman
systemctl --user restart dbus

this is on btrfs so I changed /etc/containers/storage.conf accordingly. Also the default which is "overlay" is not supported by the kernel.

# Default Storage Driver, Must be set for proper operation.
driver = "btrfs"

so far so good. Next: https://github.com/containers/podman/blob/master/docs/tutorials/rootless_tutorial.md

usermod --add-subuids 200000-265536 --add-subgids 200000-265536 user

podman unshare cat /proc/self/uid_map
         0       1000          1
         1     200000      65536

Now pull is working

podman pull bitnami/minideb:latest
Resolved "bitnami/minideb" as an alias (/home/user/.config/.cache/containers/short-name-aliases.conf)
Trying to pull docker.io/bitnami/minideb:latest...
Getting image source signatures
Copying blob 133717132a92 done  
Copying config 2f4357dd96 done  
Writing manifest to image destination
Storing signatures
2f4357dd9647eb9ebab7c0763d46e75897cacdee9bbf41e2b9f3031759355b5f

But unfortunately I get a permission denied when trying to start the container

podman run --rm 2f4357dd9647 /bin/echo "fubar"
Error: mount `proc` to `/proc`: Operation not permitted: OCI permission denied

Debug Output:

DEBU[0000] running conmon: /usr/libexec/podman/conmon    args="[--api-version 1 -c 75d6a9a73eec946697fc8d0409da9eb086fc697a62f236ce8da11cc87c5e2937 -u 75d6a9a73eec946697fc8d0409da9eb086fc697a62f236ce8da11cc87c5e2937 -r /usr/bin/crun -b /home/user/.local/share/containers/storage/overlay-containers/75d6a9a73eec946697fc8d0409da9eb086fc697a62f236ce8da11cc87c5e2937/userdata -p /run/user/1000/containers/overlay-containers/75d6a9a73eec946697fc8d0409da9eb086fc697a62f236ce8da11cc87c5e2937/userdata/pidfile -n compassionate_bardeen --exit-dir /run/user/1000/libpod/tmp/exits --socket-dir-path /run/user/1000/libpod/tmp/socket -l k8s-file:/home/user/.local/share/containers/storage/overlay-containers/75d6a9a73eec946697fc8d0409da9eb086fc697a62f236ce8da11cc87c5e2937/userdata/ctr.log --log-level debug --syslog --conmon-pidfile /run/user/1000/containers/overlay-containers/75d6a9a73eec946697fc8d0409da9eb086fc697a62f236ce8da11cc87c5e2937/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /home/user/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1000/containers --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg cgroupfs --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/usr/bin/fuse-overlayfs --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg --syslog --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg 75d6a9a73eec946697fc8d0409da9eb086fc697a62f236ce8da11cc87c5e2937]"
[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

DEBU[0000] Received: -1                                 
DEBU[0000] Cleaning up container 75d6a9a73eec946697fc8d0409da9eb086fc697a62f236ce8da11cc87c5e2937 
DEBU[0000] Tearing down network namespace at /run/user/1000/netns/cni-eb2e67ca-8562-bfc2-41aa-8c20afd73312 for container 75d6a9a73eec946697fc8d0409da9eb086fc697a62f236ce8da11cc87c5e2937 
DEBU[0000] unmounted container "75d6a9a73eec946697fc8d0409da9eb086fc697a62f236ce8da11cc87c5e2937" 
DEBU[0000] Removing container 75d6a9a73eec946697fc8d0409da9eb086fc697a62f236ce8da11cc87c5e2937 
DEBU[0000] Removing all exec sessions for container 75d6a9a73eec946697fc8d0409da9eb086fc697a62f236ce8da11cc87c5e2937 
DEBU[0000] Cleaning up container 75d6a9a73eec946697fc8d0409da9eb086fc697a62f236ce8da11cc87c5e2937 
DEBU[0000] Network is already cleaned up, skipping...   
DEBU[0000] Container 75d6a9a73eec946697fc8d0409da9eb086fc697a62f236ce8da11cc87c5e2937 storage is already unmounted, skipping... 
DEBU[0000] Container 75d6a9a73eec946697fc8d0409da9eb086fc697a62f236ce8da11cc87c5e2937 storage is already unmounted, skipping... 
DEBU[0000] ExitCode msg: "mount `proc` to `/proc`: operation not permitted: oci permission denied" 
Error: mount `proc` to `/proc`: Operation not permitted: OCI permission denied

Output of podman version:

Version:      3.0.1
API Version:  3.0.0
Go Version:   go1.14
Built:        Thu Jan  1 01:00:00 1970
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.19.4
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.27, commit: '
  cpus: 8
  distribution:
    distribution: debian
    version: "10"
  eventLogger: journald
  hostname: penguin
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 200000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 200000
      size: 65536
  kernel: 5.4.88-12224-gf05236dbdecf
  linkmode: dynamic
  memFree: 6968885248
  memTotal: 7008088064
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version 0.18.1-7931a-dirty
      commit: 7931a1eab0590eff4041c1f74e2844b297c31cea
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    selinuxEnabled: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.1.8
      commit: unknown
      libslirp: 4.3.1-git
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.4.4
  swapFree: 0
  swapTotal: 0
  uptime: 6h 30m 35s (Approximately 0.25 days)
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /home/user/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: 'fuse-overlayfs: /usr/bin/fuse-overlayfs'
      Version: |-
        fusermount3 version: 3.4.1
        fuse-overlayfs: version 1.4
        FUSE library version 3.4.1
        using FUSE kernel interface version 7.27
  graphRoot: /home/user/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  volumePath: /home/user/.local/share/containers/storage/volumes
version:
  APIVersion: 3.0.0
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.14
  OsArch: linux/amd64
  Version: 3.0.1

Package info (e.g. output of rpm -q podman or apt list podman):

Listing... Done
podman/unknown 100:3.0.1-2 amd64
podman/unknown 100:3.0.1-2 arm64
podman/unknown 100:3.0.1-2 armhf
podman/unknown 100:3.0.1-2 ppc64el
podman/unknown 100:3.0.1-2 s390x

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

[giuseppe]

can you show the output of findmnt -R /proc on the host?

Also, does the command unshare -pfr work for you?

$ unshare -pfr --mount-proc=/proc echo hi
hi

[ctophs]

findmnt -R /proc

TARGET                     SOURCE                    FSTYPE     OPTIONS
/proc                      proc                      proc       rw,nosuid,nodev,noexec,relatime
├─/proc/cpuinfo            lxcfs[/proc/cpuinfo]      fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other
├─/proc/diskstats          lxcfs[/proc/diskstats]    fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other
├─/proc/meminfo            lxcfs[/proc/meminfo]      fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other
├─/proc/stat               lxcfs[/proc/stat]         fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other
├─/proc/uptime             lxcfs[/proc/uptime]       fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other
└─/proc/sys/fs/binfmt_misc proc[/sys/fs/binfmt_misc] proc       rw,nosuid,nodev,noexec,relatime

unfortunately: unshare -pfr --mount-proc=/proc echo hi

unshare: mount /proc failed: Operation not permitted

am I doomed?

[giuseppe]

short answer: probably yes, depending if you can skip the lxcfs mounts.

longer answer: you need a /proc mount to be fully visible before you can mount a new proc. You will need to unmount all the lxcfs mounts on top of /proc before. This limitation in the kernel is to avoid exposing files from proc that were previously covered by another mount

I am going to close the issue because there is nothing we can do from Podman, but feel free to keep the discussion going

[ctophs]

hm I can umount those but..

❯ podman run --rm 2f4357dd9647 /bin/echo "fubar"
Error: mount `proc` to `/proc`: Operation not permitted: OCI permission denied

~ 
❯ findmnt -R /proc
TARGET SOURCE FSTYPE OPTIONS
/proc  proc   proc   rw,nosuid,nodev,noexec,relatime

~ 
❯ findmnt -R /proc
TARGET SOURCE FSTYPE OPTIONS
/proc  proc   proc   rw,nosuid,nodev,noexec,relatime

~ 
❯ unshare -pfr --mount-proc=/proc echo hi 
hi

no luck

[giuseppe]

reopened.

can do do podman unshare and check if the mounts are still there? Could you do the umount in the unshare environment as well?

when running as rootless, podman creates a separate user+mount namespace.

[ctophs]

wish I could.

❯ podman unshare

user in ~ 
❯ findmnt -R /proc
TARGET                     SOURCE                    FSTYPE     OPTIONS
/proc                      proc                      proc       rw,nosuid,nodev,noexec,relatime
├─/proc/cpuinfo            lxcfs[/proc/cpuinfo]      fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other
├─/proc/diskstats          lxcfs[/proc/diskstats]    fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other
├─/proc/meminfo            lxcfs[/proc/meminfo]      fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other
├─/proc/stat               lxcfs[/proc/stat]         fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other
├─/proc/uptime             lxcfs[/proc/uptime]       fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other
└─/proc/sys/fs/binfmt_misc proc[/sys/fs/binfmt_misc] proc       rw,nosuid,nodev,noexec,relatime

user in ~ 
❯ umount /proc/cpuinfo
umount: /proc/cpuinfo: not mounted.

[giuseppe]

another possibility is to do podman system migrate, it will destroy the rootless userns+mount namespace and recreate them

[ctophs]

Great! That probably worked.

❯ podman system migrate
❯ podman unshare

user in ~ 
❯ findmnt -R /proc
TARGET SOURCE FSTYPE OPTIONS
/proc  proc   proc   rw,nosuid,nodev,noexec,relatime

❯ exit

❯ podman run --rm 2f4357dd9647 /bin/echo "fubar"
fubar

[ctophs]

For the purpose of documentation.

Found a bug report for chrome os for this: https://bugs.chromium.org/p/chromium/issues/detail?id=1087937

so maybe this gets addressed in the future by google as default but in the meantime just hit alt+ctrl+t to open crosh then open termina to get into the vm using

vsh termina

and here enter

 lxc config set penguin security.nesting true

and restart the lxc container

lxc restart penguin

the mount points below /proc will still be there but podman won't get a permission denied anymore.