"Error: crun: mount proc to proc: Operation not permitted: OCI permission denied" in nested rootless container

[deliciouslytyped]

Issue Description

I have way too many terminals open with various facets of this, and I'm currently not set up to reproduce and run this, so I will have to come back and fill in the details properly when I get around to reproing.

These may be related:
#10864
#9813

I have the following error:

DEBU[0000] running conmon: /nix/store/1y223352pm0laijhsn7lca66qbnrg78h-conmon-2.1.7/bin/conmon  args="[--api-version 1 -c 86523cd65961b2bb37cdd671011d1b22ffcf8b73ec91d62e562695dc56f3e55d -u 86523cd65961b2bb37cdd671011d1b22ffcf8b73ec91d62e
562695dc56f3e55d -r /nix/store/54sfyqak7sn57z9gdhx4dydj7564lh6b-crun-1.8.4/bin/crun -b /home/podman/.local/share/containers/storage/overlay-containers/86523cd65961b2bb37cdd671011d1b22ffcf8b73ec91d62e562695dc56f3e55d/userdata -p /tmp/conta
iners-user-1000/containers/overlay-containers/86523cd65961b2bb37cdd671011d1b22ffcf8b73ec91d62e562695dc56f3e55d/userdata/pidfile -n modest_beaver --exit-dir /run/user/1000/libpod/tmp/exits --full-attach -s -l journald --log-level debug --s
yslog --conmon-pidfile /tmp/containers-user-1000/containers/overlay-containers/86523cd65961b2bb37cdd671011d1b22ffcf8b73ec91d62e562695dc56f3e55d/userdata/conmon.pid --exit-command /nix/store/mvgac2vn22hik73p557z7i25na83xdch-podman-4.5.0/bi
n/.podman-wrapped --exit-command-arg --root --exit-command-arg /home/podman/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /tmp/containers-user-1000/containers --exit-command-arg --log-level --exit-command
-arg debug --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --network-config-dir --exit-command-arg  --exit-command-arg --network-ba
ckend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /home/podman/.local/share/containers/storage/volumes --exit-command-arg --db-backend --exit-command-arg boltdb --exit-command-arg --transient-store=false
 --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg --syslog --exit-command-arg container 
--exit-command-arg cleanup --exit-command-arg 86523cd65961b2bb37cdd671011d1b22ffcf8b73ec91d62e562695dc56f3e55d]"
INFO[0000] Running conmon under slice user.slice and unitName libpod-conmon-86523cd65961b2bb37cdd671011d1b22ffcf8b73ec91d62e562695dc56f3e55d.scope 
[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied                                                                                                                                                                    
                                                                                                                       
DEBU[0000] Received: -1                                                                                                                                                                                                                       
DEBU[0000] Cleaning up container 86523cd65961b2bb37cdd671011d1b22ffcf8b73ec91d62e562695dc56f3e55d                                                                                                                                             
DEBU[0000] Tearing down network namespace at /run/user/1000/netns/netns-0993e2ab-0b96-bce8-6ce2-1cbffdfc2a3e for container 86523cd65961b2bb37cdd671011d1b22ffcf8b73ec91d62e562695dc56f3e55d                                                   
DEBU[0000] Unmounted container "86523cd65961b2bb37cdd671011d1b22ffcf8b73ec91d62e562695dc56f3e55d"                                                                                                                                             
DEBU[0000] ExitCode msg: "crun: mount `proc` to `proc`: operation not permitted: oci permission denied"                                                                                                                                       
Error: crun: mount `proc` to `proc`: Operation not permitted: OCI permission denied                                                                                                                                                           
DEBU[0000] Shutting down engines                                                                                       

This is, IIRC, running rootless podman in rootless podman.
The external podman version is 4.5.0 and the internal is 4.7.0 .
The internal podman is I think, using (I had to do some extra configuration of the container I'm using) systemd cgroup management.

Findmnt shows the following, for /proc in a container that I think should be similar:

├─/proc                    proc                                                                proc   rw,nosuid,nodev,noexec,relatime
│ ├─/proc/acpi             tmpfs                                                               tmpfs  ro,relatime,size=0k,uid=100000,gid=100000
│ ├─/proc/kcore            devtmpfs[/null]                                                     devtmp ro,nosuid,size=196376k,nr_inodes=488703,mode=755
│ ├─/proc/keys             devtmpfs[/null]                                                     devtmp ro,nosuid,size=196376k,nr_inodes=488703,mode=755
│ ├─/proc/timer_list       devtmpfs[/null]                                                     devtmp ro,nosuid,size=196376k,nr_inodes=488703,mode=755
│ ├─/proc/scsi             tmpfs                                                               tmpfs  ro,relatime,size=0k,uid=100000,gid=100000
│ ├─/proc/bus              proc[/bus]                                                          proc   ro,nosuid,nodev,noexec,relatime
│ ├─/proc/fs               proc[/fs]                                                           proc   ro,nosuid,nodev,noexec,relatime
│ ├─/proc/irq              proc[/irq]                                                          proc   ro,nosuid,nodev,noexec,relatime
│ ├─/proc/sys              proc[/sys]                                                          proc   ro,nosuid,nodev,noexec,relatime
│ └─/proc/sysrq-trigger    proc[/sysrq-trigger]                                                proc   ro,nosuid,nodev,noexec,relatime

I read something in the mentioned possibly related issues that proc can't be mounted over if it has mounts in it. I didn't find the kernel documentation for this.

I don't quite understand what is supposed to be happening - this may help diagnose.

• Why is podman trying to mount proc? (I'm assuming to set up a pid namespace)
• What would this look like on a system where it works? (I suppose it would help if I tried comparing against a standard container image)

### Steps to reproduce the issue

TODO

Describe the results you received

Describe the results you received

Describe the results you expected

Describe the results you expected

podman info output

TODO

Podman in a container

Yes

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

TODO

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[rhatdan]

Usually this is because you are trying to run a container within a container. If on the outer container you do.

podman run --security-opt unmask=/proc ...
Then the issue inside of the container should go away.

[deliciouslytyped]

Hm. This isn't an SELinux thing right? I'm not using SELinux. What is masking?
And thanks, I'll try this out later.

[rhatdan]

No this is not an SELinux issue. When podman or any container engine creates a container it masks over sections of /proc, and then within the container if you run another container engine, that container engine attempts to also modify /proc, and basically the kernel does not allow modification of a modified /proc. The --security-opt unmask=/proc, tells the outer container engine to not modify /proc.

[giuseppe]

@deliciouslytyped does --security-opt unmask=/proc/* solve your problem?

[deliciouslytyped]

Sorry, I haven't had time to look at this. I'll likely wont have time for a few weeks, we'll see. Maybe over the weekend if I can repro the issue quickly.

[deliciouslytyped]

For what it's worth I run into a similar looking issue with buildah, but that doesn't have any unmask flags;

error running container: from crun creating container for [/bin/sh -c pip install --trusted-host pypi.python.org -r django]: mount `proc` to `proc`: Operation not permitted

Edit: forgot that you stated that the unmask needs to be done on the outer container, which makes sense.

[deliciouslytyped]

Unmasking appears to have helped with what I'm trying currently. I still need to get back to what the issue is originally about. 👍

[giuseppe]

I am closing the issue since the error is expected with a masked proc

[raqqun]

I have the same error using podman inside a kubernetes container.

I use buildah as part of the ci routine to build the image then i use podman to do some testing. Everything is run on the same ci container. buildah is fine it builds the image correctly bud when i try to podman run ... i get Error: crun: mount 'proc' to 'proc': Operation not permitted: OCI permission denied.

findmnt -R /proc says:

 + findmnt -R /proc
12:54:48  TARGET                SOURCE               FSTYPE OPTIONS
12:54:48  /proc                 proc                 proc   rw,nosuid,nodev,noexec,relatime
12:54:48  ├─/proc/bus           proc[/bus]           proc   ro,nosuid,nodev,noexec,relatime
12:54:48  ├─/proc/fs            proc[/fs]            proc   ro,nosuid,nodev,noexec,relatime
12:54:48  ├─/proc/irq           proc[/irq]           proc   ro,nosuid,nodev,noexec,relatime
12:54:48  ├─/proc/sys           proc[/sys]           proc   ro,nosuid,nodev,noexec,relatime
12:54:48  ├─/proc/sysrq-trigger proc[/sysrq-trigger] proc   ro,nosuid,nodev,noexec,relatime
12:54:48  ├─/proc/acpi          tmpfs                tmpfs  ro,relatime,context="system_u:object_r:data_t:s0:c616,c956"
12:54:48  ├─/proc/kcore         tmpfs[/null]         tmpfs  rw,nosuid,context="system_u:object_r:data_t:s0:c616,c956",size=65536k,mode=755
12:54:48  ├─/proc/keys          tmpfs[/null]         tmpfs  rw,nosuid,context="system_u:object_r:data_t:s0:c616,c956",size=65536k,mode=755
12:54:48  ├─/proc/latency_stats tmpfs[/null]         tmpfs  rw,nosuid,context="system_u:object_r:data_t:s0:c616,c956",size=65536k,mode=755
12:54:48  ├─/proc/timer_list    tmpfs[/null]         tmpfs  rw,nosuid,context="system_u:object_r:data_t:s0:c616,c956",size=65536k,mode=755
12:54:48  └─/proc/scsi          tmpfs                tmpfs  ro,relatime,context="system_u:object_r:data_t:s0:c616,c956"

podman info:

15:32:56  + podman info
15:32:56  host:
15:32:56    arch: amd64
15:32:56    buildahVersion: 1.28.2
15:32:56    cgroupControllers:
15:32:56    - cpuset
15:32:56    - cpu
15:32:56    - io
15:32:56    - memory
15:32:56    - hugetlb
15:32:56    - pids
15:32:56    - misc
15:32:56    cgroupManager: cgroupfs
15:32:56    cgroupVersion: v2
15:32:56    conmon:
15:32:56      package: conmon_2.1.6+ds1-1_amd64
15:32:56      path: /usr/bin/conmon
15:32:56      version: 'conmon version 2.1.6, commit: unknown'
15:32:56    cpuUtilization:
15:32:56      idlePercent: 73.06
15:32:56      systemPercent: 2.39
15:32:56      userPercent: 24.55
15:32:56    cpus: 8
15:32:56    distribution:
15:32:56      codename: bookworm
15:32:56      distribution: debian
15:32:56      version: "12"
15:32:56    eventLogger: file
15:32:56    hostname: builder-build-custom-lfe-jenkins-image-2-426-2-ocean-2-15-gx4sp
15:32:56    idMappings:
15:32:56      gidmap:
15:32:56      - container_id: 0
15:32:56        host_id: 10000
15:32:56        size: 1
15:32:56      - container_id: 1
15:32:56        host_id: 10001
15:32:56        size: 65536
15:32:56      uidmap:
15:32:56      - container_id: 0
15:32:56        host_id: 10000
15:32:56        size: 1
15:32:56      - container_id: 1
15:32:56        host_id: 10001
15:32:56        size: 65536
15:32:56    kernel: 5.15.139
15:32:56    linkmode: dynamic
15:32:56    logDriver: k8s-file
15:32:56    memFree: 18642423808
15:32:56    memTotal: 33084821504
15:32:56    networkBackend: cni
15:32:56    ociRuntime:
15:32:56      name: crun
15:32:56      package: crun_1.8.1-1+deb12u1_amd64
15:32:56      path: /usr/bin/crun
15:32:56      version: |-
15:32:56        crun version 1.8.1
15:32:56        commit: f8a096be060b22ccd3d5f3ebe44108517fbf6c30
15:32:56        rundir: /tmp/podman-run-10000/crun
15:32:56        spec: 1.0.0
15:32:56        +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
15:32:56    os: linux
15:32:56    remoteSocket:
15:32:56      path: /tmp/podman-run-10000/podman/podman.sock
15:32:56    security:
15:32:56      apparmorEnabled: false
15:32:56      capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
15:32:56      rootless: true
15:32:56      seccompEnabled: true
15:32:56      seccompProfilePath: /usr/share/containers/seccomp.json
15:32:56      selinuxEnabled: false
15:32:56    serviceIsRemote: false
15:32:56    slirp4netns:
15:32:56      executable: /usr/bin/slirp4netns
15:32:56      package: slirp4netns_1.2.0-1_amd64
15:32:56      version: |-
15:32:56        slirp4netns version 1.2.0
15:32:56        commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
15:32:56        libslirp: 4.7.0
15:32:56        SLIRP_CONFIG_VERSION_MAX: 4
15:32:56        libseccomp: 2.5.4
15:32:56    swapFree: 0
15:32:56    swapTotal: 0
15:32:56    uptime: 12h 32m 23.00s (Approximately 0.50 days)
15:32:56  plugins:
15:32:56    authorization: null
15:32:56    log:
15:32:56    - k8s-file
15:32:56    - none
15:32:56    - passthrough
15:32:56    - journald
15:32:56    network:
15:32:56    - bridge
15:32:56    - macvlan
15:32:56    - ipvlan
15:32:56    volume:
15:32:56    - local
15:32:56  registries: {}
15:32:56  store:
15:32:56    configFile: /home/jenkins/.config/containers/storage.conf
15:32:56    containerStore:
15:32:56      number: 0
15:32:56      paused: 0
15:32:56      running: 0
15:32:56      stopped: 0
15:32:56    graphDriverName: overlay
15:32:56    graphOptions: {}
15:32:56    graphRoot: /home/jenkins/.local/share/containers/storage
15:32:56    graphRootAllocated: 105666383872
15:32:56    graphRootUsed: 36953112576
15:32:56    graphStatus:
15:32:56      Backing Filesystem: extfs
15:32:56      Native Overlay Diff: "true"
15:32:56      Supports d_type: "true"
15:32:56      Using metacopy: "false"
15:32:56    imageCopyTmpDir: /var/tmp
15:32:56    imageStore:
15:32:56      number: 5
15:32:56    runRoot: /tmp/containers-user-10000/containers
15:32:56    volumePath: /home/jenkins/.local/share/containers/storage/volumes
15:32:56  version:
15:32:56    APIVersion: 4.3.1
15:32:56    Built: 0
15:32:56    BuiltTime: Thu Jan  1 00:00:00 1970
15:32:56    GitCommit: ""
15:32:56    GoVersion: go1.19.8
15:32:56    Os: linux
15:32:56    OsArch: linux/amd64
15:32:56    Version: 4.3.1

buildah info:

15:32:56  {
15:32:56      "host": {
15:32:56          "CgroupVersion": "v2",
15:32:56          "Distribution": {
15:32:56              "distribution": "debian",
15:32:56              "version": "12"
15:32:56          },
15:32:56          "MemFree": 18647769088,
15:32:56          "MemTotal": 33084821504,
15:32:56          "OCIRuntime": "crun",
15:32:56          "SwapFree": 0,
15:32:56          "SwapTotal": 0,
15:32:56          "arch": "amd64",
15:32:56          "cpus": 8,
15:32:56          "hostname": "builder-build-custom-lfe-jenkins-image-2-426-2-ocean-2-15-gx4sp",
15:32:56          "kernel": "5.15.139",
15:32:56          "os": "linux",
15:32:56          "rootless": true,
15:32:56          "uptime": "12h 32m 23.68s (Approximately 0.50 days)",
15:32:56          "variant": ""
15:32:56      },
15:32:56      "store": {
15:32:56          "ContainerStore": {
15:32:56              "number": 0
15:32:56          },
15:32:56          "GraphDriverName": "overlay",
15:32:56          "GraphOptions": null,
15:32:56          "GraphRoot": "/home/jenkins/.local/share/containers/storage",
15:32:56          "GraphStatus": {
15:32:56              "Backing Filesystem": "extfs",
15:32:56              "Native Overlay Diff": "true",
15:32:56              "Supports d_type": "true",
15:32:56              "Using metacopy": "false"
15:32:56          },
15:32:56          "ImageStore": {
15:32:56              "number": 5
15:32:56          },
15:32:56          "RunRoot": "/var/tmp/containers-user-10000/containers"
15:32:56      }
15:32:56  }

I've tested the proposed solutions here and there at differents issues, nothing seems to work.

One important think is that kubernetes is running on bottlerocket-os with eks, any idea on that ? maybe a kernel lockdown at the host level ? Basically the idea is to be able to run unprivileged containers and run podman tests through testcontainers an other testing tools.

[rhatdan]

If /proc inside of the k8s container has been modified, then Podman will not be able to modify it. Kernel enforces this. Bottom line you have to tell k8s to not modify /proc which is being mounted into the container.

[raqqun]

How would /proc be modified inside the container ? Does buildah modify it during building ? And how to tell k8s "to not modify /proc" i didn't undestand that correctly.

Also another think i didn't quite understand is how can buildah run the RUN commands during build and podman can't do the same ?

[rhatdan]

Are you running with --isolation=chroot?

YOu could try to run with --pid=host inside of container.

[raqqun]

Yes thank you very much. I spend two todays reviewing podman documentation and repository and i tried to iterate my initial podman configuration many times to understand how podman works.

This is the final result :

[containers]
netns="host"
userns="host"
ipcns="host"
utsns="host"
cgroupns="host"
pidns="host"
cgroups="enabled"
log_driver = "k8s-file"
[engine]
cgroup_manager = "cgroupfs"
events_logger="file"
runtime="crun"

I wanted to test how far things can go and i stopped at three levels of nested podmans inside a kubernetes container which is already incredible.

[raqqun]

The last think now that i can't undestand is why do i have to :

  RUN chmod u-s /usr/bin/new[gu]idmap \
      && setcap cap_setuid=ep /usr/bin/newuidmap \
      && setcap cap_setgid=ep /usr/bin/newgidmap

Without that i get : newuidmap: write to uid_map failed: Operation not permitted

Any ideas as to why podman doesn't like that sticky bit on newuidmap binary (it comes by default on debian based with uidmap package) ?

[deliciouslytyped]

That's not sticky that's suid no?
I've run into similar but I don't think I ever exactly figured it out.
Maybe it has to do with the nested container user being root or not.

[rhatdan]

This is an issue with the fedora base image not setting up the "filecap" bit correctly.

[deliciouslytyped]

This is explained in other issues (don't have a link on hand) - but what is it about the filecap not being set correctly that actually results in the problem? (possibly explained in the aforementioned )

[rhatdan]

The file cap is like a setuid setting, that allows newuidmap and newgidmap to configure the usernamespace based on the contents of /etc/subuid and /etc/subgid. This is a privileged operation. No file caps and rootless Podman no longer works.

[raqqun]

@rhatdan According to this PR https://github.com/containers/storage/pull/1188/files containers/storage checks for both setuid and filecap but for whatever reasons this doesn't work. The Debian package uidmap sets setuid on both new[u,g]idmap but on run time you get permission denied on both. My question is more oriented on podman's ability to play with setuid (basicaly chmod u+s) or if podman prefers filecaps over legacy setuid.

[rhatdan]

Podman does nothing other then execute the programs, it is up to the kernel to enforce these bits. A couple of questions though the kernel will not allow setuid of filecaps on file systems mounted with NOSUID (Rootless homedir mounted NOSUID), and if you are running code in emulation then the kernel will also ignore them. Do either of these make sense for your case?

[raqqun]

Yes it sounds like bottlerocketos enforces this. Thanx a lot. I'll have to do some digging on botlerocket.

[TristanCacqueray]

If /proc inside of the k8s container has been modified, then Podman will not be able to modify it. Kernel enforces this. Bottom line you have to tell k8s to not modify /proc which is being mounted into the container.

Wouldn't it be possible to keep the /proc modifications done by k8s while still providing a fresh pid namespaced proc mount? In our case, we are not able to run nested container when there is a tmpfs mounted on /proc/scsi.

[rhatdan]

Try it but I doubt it.

[TristanCacqueray]

@rhatdan from the host I observe that podman is able to create a fresh procfs when it is modified, while buildah/unshare is not able to, would you know how this works?

Here is a reproducer on fedora-39:

# Modify /proc
[tristanc@fedora ~]$ sudo mount -t tmpfs none /proc/scsi

# buildah: KO
[tristanc@fedora ~]$ buildah run $(buildah from scratch) id
2024-02-16T20:39:26.516687Z: mount `proc` to `proc`: Operation not permitted

# unshare: KO
[tristanc@fedora ~]$ unshare --user --mount --net --pid --fork --map-root-user --mount-proc
unshare: mount /proc failed: Operation not permitted

# podman: OK
[tristanc@fedora ~]$ podman run --rm fedora ps afx
    PID TTY      STAT   TIME COMMAND
      1 ?        Rs     0:00 ps afx

# Cleanup
[tristanc@fedora podman]$ sudo umount /proc/scsi

[giuseppe]

Wouldn't it be possible to keep the /proc modifications done by k8s while still providing a fresh pid namespaced proc mount? In our case, we are not able to run nested container when there is a tmpfs mounted on /proc/scsi.

no, it is not possible. You must have a fully visible proc file system before you can mount a new one in a user namespace.

We need this in Kubernetes: kubernetes/enhancements#4265

[rhatdan]

@giuseppe why is Podman successful, but buildah not? I would have thought podman would have blown up the same way as buildah in the example above?

[giuseppe]

@giuseppe why is Podman successful, but buildah not? I would have thought podman would have blown up the same way as buildah in the example above?

podman joins the existing user+mount namespace. If you run podman system migrate after the sudo mount -t tmpfs none /proc/scsi command, you'll see the same failure