Can't adjust fs.mqueue.* sysctls inside rootless container

[jkisiele]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Hello,
Let me preface everything with me saying that I'm not sure this is a bug or just me lacking experience and going against conventional wisdom. If that's the case - forgive me.

I'm trying to create a container as a rootless user with custom values for kernel parameters regarding message queues (fs.mqueue.*) with the --sysctl switch but I'm unable to do so.
The host machine running podman is a stock Fedora 31 system with no configuration file change.

The podman run command listed below runs successfuly when executed with sudo but I need it to work without root privileges.

Steps to reproduce the issue:

podman run --rm --name sysctl-test --sysctl fs.mqueue.msg_max=20 docker.io/library/centos

Describe the results you received:

Error: open /proc/sys/fs/mqueue/msg_max: Permission denied: OCI runtime permission denied error

Describe the results you expected:

Successful creation and startup of a CentOS container with modified fs.mqueue.msg_max value

Additional information you deem important (e.g. issue happens only occasionally):

Aforementioned error seems to appear only in regards to the fs.mqueue.* parameters.
I've tried modifying other parameters which are listed as namespaced in the podman documentation with the --sysctl switch and they were successful

Example:
podman run --rm --name sysctl-test --sysctl net.ipv4.ip_forward=0 docker.io/library/centos

Output of podman version:

Version:            1.8.0
RemoteAPI Version:  1
Go Version:         go1.13.6
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.13.6
  podman version: 1.8.0
host:
  BuildahVersion: 1.13.1
  CgroupVersion: v2
  Conmon:
    package: conmon-2.0.10-2.fc31.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.10, commit: 6b526d9888abb86b9e7de7dfdeec0da98ad32ee0'
  Distribution:
    distribution: fedora
    version: "31"
  IDMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  MemFree: 1229598720
  MemTotal: 8029335552
  OCIRuntime:
    name: crun
    package: crun-0.12.2.1-1.fc31.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.12.2.1
      commit: cd7cea7114db5f6aa35fbb69fa307c19c2728a31
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  SwapFree: 8170500096
  SwapTotal: 8170500096
  arch: amd64
  cpus: 8
  eventlogger: journald
  hostname: 15z990
  kernel: 5.5.6-201.fc31.x86_64
  os: linux
  rootless: true
  slirp4netns:
    Executable: /usr/bin/slirp4netns
    Package: slirp4netns-0.4.0-20.1.dev.gitbbd6f25.fc31.x86_64
    Version: |-
      slirp4netns version 0.4.0-beta.3+dev
      commit: bbd6f25c70d5db2a1cd3bfb0416a8db99a75ed7e
  uptime: 24h 28m 7.56s (Approximately 1.00 days)
registries:
  search:
  - docker.io
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - quay.io
store:
  ConfigFile: /home/jakub/.config/containers/storage.conf
  ContainerStore:
    number: 0
  GraphDriverName: overlay
  GraphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-0.7.5-2.fc31.x86_64
      Version: |-
        fusermount3 version: 3.6.2
        fuse-overlayfs: version 0.7.5
        FUSE library version 3.6.2
        using FUSE kernel interface version 7.29
  GraphRoot: /home/jakub/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 2
  RunRoot: /run/user/1000/containers
  VolumePath: /home/jakub/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

podman-1.8.0-2.fc31.x86_64

[rhatdan]

Works for me on f32?
Could this be an SELinux issue?

[jkisiele]

@rhatdan Thank you for the suggestion, but setting SELinux to Permissive was the first thing that I've tried. I should've mentioned it in the OP - sorry.

[jakub@15z990 ~]$ getenforce
Permissive
[jakub@15z990 ~]$ podman run --rm --name sysctl-test --sysctl fs.mqueue.msg_max=20 docker.io/library/centos
Error: open /proc/sys/fs/mqueue/msg_max: Permission denied: OCI runtime permission denied error

[rhatdan]

Try with --privileged, to see if it is dropped caps or seccomp.
If this works we can dig deeper into what is going on.
If not, then I would guess it is something to do with User Namespace.

[jkisiele]

Tried as instructed (with SELinux in Enforcing and Permissive mode), got the same result.

[jakub@15z990 ~]$ podman run --rm --name sysctl-test --privileged=true --sysctl fs.mqueue.msg_max=20 docker.io/library/centos
Error: open /proc/sys/fs/mqueue/msg_max: Permission denied: OCI runtime permission denied error

[rhatdan]

Ok it looks like user namespace then.

podman unshare cat /proc/sys/fs/mqueue/msg_max

[rhatdan]

$ podman run --rm docker.io/library/centos ls -l /sys/fs/mqueue/msg_max
$ podman run --rm docker.io/library/centos cat /sys/fs/mqueue/msg_max
$ podman run --rm docker.io/library/centos bash -c "echo 20 > /sys/fs/mqueue/msg_max"

[jkisiele]

Okay, I've modified the msg_max parameter to a value on the host machine to one different than default (10) to illustrate better. No idea if this will help but trying my best to make things easier for everyone involved.

[jakub@15z990 ~]$ cat /proc/sys/fs/mqueue/msg_max 
29

[jakub@15z990 ~]$ podman unshare cat /proc/sys/fs/mqueue/msg_max
29

For the last three I'm guessing there should be a /proc prefixed to the path:

[jakub@15z990 ~]$ podman run --rm docker.io/library/centos ls -l /proc/sys/fs/mqueue/msg_max
-rw-r--r--. 1 nobody nobody 0 Mar  3 21:37 /proc/sys/fs/mqueue/msg_max

[jakub@15z990 ~]$ podman run --rm docker.io/library/centos cat /proc/sys/fs/mqueue/msg_max
10

[jakub@15z990 ~]$ podman run --rm docker.io/library/centos bash -c "echo 20 > /proc/sys/fs/mqueue/msg_max"
bash: /proc/sys/fs/mqueue/msg_max: Read-only file system

The contents of my /etc/subuid and /etc/subgid were generated during installation and weren't modified:

[jakub@15z990 ~]$ cat /etc/subuid
jakub:100000:65536

[jakub@15z990 ~]$ cat /etc/subgid
jakub:100000:65536

[rhatdan]

@giuseppe Does this work for you?

[giuseppe]

I think the kernel is blocking it:

$ unshare --mount-proc --keep-caps -rmiunp -f bash
# echo 20 > /proc/sys/fs/mqueue/msg_max
bash: /proc/sys/fs/mqueue/msg_max: Permission denied

[rhatdan]

So this is a kernel issue.

[NathanaelGandhi]

I still experience this issue with podman. I am able to set --sysctl fs.mqueue.msg_max=50 correctly with docker

[giuseppe]

Are you running podman as root user?

[NathanaelGandhi]

no, I am running rootless as per the setup instructions: https://wiki.archlinux.org/title/Podman#Rootless_Podman

Using the example mentioned in this issue

podman run --rm --name sysctl-test --sysctl fs.mqueue.msg_max=20 docker.io/library/centos 
Error: crun: open /proc/sys/fs/mqueue/msg_max: Permission denied: OCI permission denied

A proposed solution is mentioned: https://bbs.archlinux.org/viewtopic.php?id=253966 and #4162 however setting DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus appears to not to work

I have also tried

• adding --privileged=true
• podman unshare cat /proc/sys/fs/mqueue/msg_max
• podman run --rm docker.io/library/centos ls -l /sys/fs/mqueue/msg_max
• podman run --rm docker.io/library/centos cat /sys/fs/mqueue/msg_max
• podman run --rm docker.io/library/centos bash -c "echo 20 > /sys/fs/mqueue/msg_max"

[giuseppe]

an unprivileged user cannot change that value. You need to be root on the host.

Said so, with some manual hacks, root on the host could change that setting for the unprivileged container as well, but this must be done after the container already runs and the namespaces are created (you need to use nsenter as root to join the container namespaces).