Fail to run privileged nested container, on fedora 41

[Yarboa]

Issue Description

Describe your issue

Fail to run privileged nested container, on fedora 41

Steps to reproduce the issue

Steps to reproduce the issue

1. Fedora41
2. dnf copr enable -y @centos-automotive-sig/bluechi-snapshot "Fedora-41-$(arch)"
3. dnf copr enable -y rhcontainerbot/qm "fedora-41-$(arch)"
4. dnf install -y qm
5. Now, need to run emulated setup script to install qm systemd and podman in QM rootfs.
   bash -x /usr/share/qm/setup
6. Verify qm service is up

    systemctl is-acive qm 
    podman ps shows qm enviroment is running.

7. Run nested container, rootfull

    podman exec qm podman run ubi9-minimal pwd

Describe the results you received

Describe the results you received

    Resolved "ubi9-minimal" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
    Trying to pull registry.access.redhat.com/ubi9-minimal:latest...
    Getting image source signatures
    Writing manifest to image destination
    Storing signatures
Error: crun: setrlimit `RLIMIT_NOFILE`: Operation not permitted: OCI permission denied

Describe the results you expected

Describe the results you expected
Nested container should run without failures.host-podman-info.txt
qm-podman-info.txt

Nested podman container is run successfully for c9s
https://artifacts.dev.testing-farm.io/0d73bf4f-14b2-43c9-8bcf-27b07f3406fb/work-kvm-tier-05glnpgbx/plans/e2e/kvm-tier-0/execute/data/guest/default-0/tests/qm-sanity-test/check_qm_podman_quadlet_is_ok-4/output.txt

podman info output

If you are unable to run podman info for any reason, please provide the podman version, operating system and its version and the architecture you are running.

*podman info*

Please see attached files
host-podman-info.txt
attached file for qm
qm-podman-info.txt

*podman version*

rpm -q podman
podman-5.3.1-1.fc41.x86_64

podman exec qm rpm -q podman
podman-5.3.1-1.fc41.x86_64


*operating system* 
Fedora
*version*
41, uname -r
6.11.8-300.fc41.x86_64

architecture
x86_64

Podman in a container

Yes

Privileged Or Rootless

Privileged

Upstream Latest Release

Yes

Additional environment details

Additional environment details

To understand tested enviroment please refer this README
https://github.com/containers/qm?tab=readme-ov-file#qm-is-a-containerized-environment-for-running-functional-safety-qm-quality-management-software

Additional information

more details exist here
qm#660

This one containers/qm#666 (comment)
Did not help to solve the issue

[Luap99]

Can you provide the actual podman commands used please.

[Yarboa]

Sure see Steps to reproduce the issue

But the command itself is step number 7

podman exec qm podman run ubi9-minimal pwd

qm is running with the following quadlet
https://github.com/containers/qm/blob/main/qm.container

The issue is nested container inside qm
When remove all PodmanArgs and use --priviliged in quadlet file nested container is forked successfully

In case you need ready env in TestingFarm, i can prepare it for you

[Luap99]

LimitNOFILE=65536

Ok I see, your outer container sets a low file limit and you do not give the container CAP_SYS_RESOURCE so it cannot increase it. Since commit
5ebba75 we try to set a static default limit. So you either need to use a higher limit for the outer container or use a different limit for the inner container, either via --ulimit or there should be a containers.conf default I think.

The alternative is for podman to not assume we can increase the limits as root (check for CAP_SYS_RESOURCE) and if not just clamp it like with the rootless case: #24508 (comment)

cc @giuseppe

[giuseppe]

Ok I see, your outer container sets a low file limit and you do not give the container CAP_SYS_RESOURCE so it cannot increase it. Since commit 5ebba75 we try to set a static default limit. So you either need to use a higher limit for the outer container or use a different limit for the inner container, either via --ulimit or there should be a containers.conf default I think.

The alternative is for podman to not assume we can increase the limits as root (check for CAP_SYS_RESOURCE) and if not just clamp it like with the rootless case: #24508 (comment)

yes good analysis.

I've opened a PR to fix it: #24696

[Yarboa]

@Luap99 thanks

Add this to quadlet drop-in

cat /etc/containers/systemd/qm.container.d/add_resource.conf 
[Container]
DropCapability=
AddCapability=sys_resource

Did the trick

So to understand the ulimit explanation
I see that in outer container i have this
open files (-n) 1024

and in internal container
open files (-n) 524288

So do i have to change one of those?
Thanks again

[Yarboa]

OK, i think i got it from here
man getrlimit | grep RLIMIT_NPROC

So, the issue is conflicting quadlet values on rlimit due to changes in podman implimatations
We have this unlimited
https://github.com/containers/qm/blob/main/qm.container#L34
https://github.com/containers/qm/blob/main/qm.container#L45

One more question here
I changed the limits under as following

host:

[root@90f2f529-751d-45bd-8bd6-5b67ac6e4e59 ~]# ulimit -n -H
524288
[root@90f2f529-751d-45bd-8bd6-5b67ac6e4e59 ~]# ulimit -n -S
1024

parent container:

podman exec -it qm bash ulimit -n -H
2048
[root@90f2f529-751d-45bd-8bd6-5b67ac6e4e59 ~]# podman exec -it qm bash ulimit -n -S
512

Running container

podman exec qm podman run --ulimit nofile=256:1024  ubi9-minimal pwd
Error: crun: setrlimit `RLIMIT_NPROC`: Operation not permitted: OCI permission denied

and --log-level debug, still return this error

time="2024-11-28T10:39:46Z" level=info msg="Running conmon under slice machine.slice and unitName libpod-conmon-ed346a440522e8ad439244fec83198af9a367ce092fa18ba185b481bfb90b229.scope"
[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

time="2024-11-28T10:39:46Z" level=debug msg="Received: -1"

Is AddCapability=sys_resource th only solution ?
Is there another alternative to that solution?

[Luap99]

[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

That can be ignored conmon tries to bump this so it is not killed before container process as we want conmon to work and correctly spawn the container cleanup command. But as rootless this is not possible so it safe to ignore.

podman exec qm podman run --ulimit nofile=256:1024 ubi9-minimal pwd
Error: crun: setrlimit RLIMIT_NPROC: Operation not permitted: OCI permission denied

That is now failing on NPROC, you fixed the NOFILE issue

With #24696 merged it should just work without any changes for you again if you test podman from main. Podman checks for sys_resource and not try to bump the limits higher when it does have the cap.