QM:podman run is failing in Fedora 41

[Yarboa]

During the work on #660 fedora QM podman run is failing with the following error while spawning rootful container inside qm

Error: crun: setrlimit `RLIMIT_NOFILE`: Operation not permitted: OCI permission denied
[ FAILED ] Error: Command podman exec qm podman run alpine echo Hello QM failed with exit code: 126

https://artifacts.dev.testing-farm.io/0b3cd818-fbdb-40a1-bcc8-d6d7e81ead39/

TODO:
Add update with extra details

uname -r
6.11.8-300.fc41.x86_64

podman infos
podman-qm-info.json
podman-host-info.json

Running podman logs, attached reveal this, podman-err.log

[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

No selinux errors

ausearch -m AVC  -ts recent
<no matches>

Note when replacing quadlet, see link, Podman args with --privileged container is forked successfully
https://github.com/containers/qm/blob/main/qm.container

Additional Info
Podman inspect log

               "Ulimits": [
                    {
                         "Name": "RLIMIT_NOFILE",
                         "Soft": 1048576,
                         "Hard": 1048576
                    }, 
                    { 
                         "Name": "RLIMIT_NPROC",
                         "Soft": 1048576,
                         "Hard": 1048576
                    }
               ],

podman-inspect.log

Ulimits

[root@505c2268-5b27-4bd8-819f-e58aa67cd028 ~]# ulimit -a -S
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) unlimited
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 31594
max locked memory           (kbytes, -l) 8192
max memory size             (kbytes, -m) unlimited
open files                          (-n) 1024
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) 8192
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) 31594
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited
[root@505c2268-5b27-4bd8-819f-e58aa67cd028 ~]# ulimit -a -H
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) unlimited
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 31594
max locked memory           (kbytes, -l) 8192
max memory size             (kbytes, -m) unlimited
open files                          (-n) 524288
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) unlimited
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) 31594
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited

[Yarboa]

@giuseppe @rhatdan Can you please take a look?

[giuseppe]

can you please try with upstream Podman? Could be fixed by containers/podman#24547

Do you have a Podman command line that shows the error? Are you running as root? Is it inside a user namespace?

[Yarboa]

can you please try with upstream Podman? Could be fixed by containers/podman#24547

Do you have a Podman command line that shows the error? Are you running as root? Is it inside a user namespace?

the command goes as following

[root@505c2268-5b27-4bd8-819f-e58aa67cd028 ~]# podman exec qm podman run ubi9-minimal pwd
Error: crun: setrlimit `RLIMIT_NOFILE`: Operation not permitted: OCI permission denied

Yes it is run as root.

Taking a look at pr24547
Do you refer this ?
https://podman.io/docs/installation#fedora-1

repo updates-testing?

[giuseppe]

thanks, that is indeed nested podman, so I think the PR I have mentioned fixes your problem

[Yarboa]

thanks, that is indeed nested podman, so I think the PR I have mentioned fixes your problem

same problem with this

dnf -y install --use-host-config   podman
Updating and loading repositories:
Repositories loaded.
Package "podman-5:5.3.1-1.fc41.x86_64" is already installed.

dnf -y install --use-host-config  --installroot /usr/lib/qm/rootfs/ podman
Updating and loading repositories:
 Copr repo for podman-next owned by rhcontainerbot    100% | 281.5 KiB/s | 139.6 KiB |  00m00s 
Repositories loaded.
Package "podman-5:5.3.1-1.fc41.x86_64" is already installed

still failing

podman exec qm podman run ubi9-minimal pwd
Error: crun: setrlimit `RLIMIT_NOFILE`: Operation not permitted: OCI permission denied
[root@475e2329-93c7-4fe6-b4bf-d7b4ea581832 ~]# 

With --log-level debug

time="2024-11-26T13:40:14Z" level=info msg="Running conmon under slice machine.slice and unitName libpod-conmon-4f17acf681a4407247b456479b905e240e2435e877deb4e9d366700de3f9bc79.scope"
[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

[giuseppe]

have you updated the podman inside the image?

[Yarboa]

have you updated the podman inside the image?

yes, it is partitioned file-system in qm repo not an image

[root@475e2329-93c7-4fe6-b4bf-d7b4ea581832 ~]# podman exec qm rpm -q podman
podman-5.3.1-1.fc41.x86_64

[giuseppe]

do you prefer if we move this issue to podman or do you create a new one?

Please provide a reproducer using just podman (both the external container, and the nested one). Please specify how the external container was created.

[Yarboa]

Sure thanks I will do that @giuseppe thanks,

[Yarboa]

@giuseppe thanks
containers/podman#24692

[Yarboa]

@dougsland based on the issue in containers.
I found this change 94998b5

So from here
man systemd-system.conf | grep LimitNOFILE=
DefaultLimitCORE=, DefaultLimitRSS=, DefaultLimitNOFILE=, DefaultLimitAS=,
• DefaultLimitNOFILE= defaults to 1024:524288.

But this value is unlimited for QM, this requires CAP_RESOURCE
We have the following quadlet mapping

DropCapability=sys_resource

Asking for advise here
@dougsland @rhatdan @alexlarsson

Summery of the issue
containers/podman#24692 (comment)

Note:
we can change this value only for prepare plan of testing with modifying this value

[Yarboa]

OK there is another option:
So we need to play with nofile=H:S nproc=H:S

Added this to quadlet drop-in

[Service]
LimitNOFILE=
OOMScoreAdjust=

[Container]
PodmanArgs=--ulimit="nofile=512:2048,nproc=14000:15000"

HOST
then run the following
[root@90f2f529-751d-45bd-8bd6-5b67ac6e4e59 ~]# ulimit -n -u -H
open files (-n) 524288
max user processes (-u) 31594
[root@90f2f529-751d-45bd-8bd6-5b67ac6e4e59 ~]# ulimit -n -u -S
open files (-n) 1024
max user processes (-u) 31594

[root@90f2f529-751d-45bd-8bd6-5b67ac6e4e59 ~]# podman exec qm ulimit -n -u -H
open files (-n) 2048
max user processes (-u) 15000
[root@90f2f529-751d-45bd-8bd6-5b67ac6e4e59 ~]# podman exec qm ulimit -n -u -S
open files (-n) 512
max user processes (-u) 14000

Now running container
```
podman exec qm podman run --ulimit nofile=256:1024,nproc=14000:15000  ubi9-minimal pwd
/
```

[alexlarsson]

So. my understanding is that systemd launches the qm container with a limit max open files, and then when podman runs in the qm container it tries to set the subcontainer max open file to what it thinks is the "default". However, this value is higher than the limit systemd set, so we fail.

The real fix is containers/podman#24696 which means podman will never try to raise the max if it doesn't have the capability to do so.

Now, possible workarounds are:

Update to the fixed podman: This is the long term fix, but may be hard atm.

Give podman the capability to raise the max in the qm container. This is obviously unwanted, we might as well just drop the max we set, as it makes it worthless, and we set it for a reason.

Change the default max in containers.conf inside the qm container so podman doesn't try to raise the max. This is what i think we should do for now.

[Yarboa]

OK, that did the trick, adding this into /etc/qm/containers/containers.conf

default_ulimits = [
"nofile=1023:1023", #-1 of host -n
"nproc=2048:2048", #50% of host -u
]