Podman in podman combinatorics

[th-hummel]

Hello,

running podman-5.2.3-1.fc40.x86_64 I'm experimenting with running podman inside podman (with quay.io/podman/stable image) in different setups among (rootful, rootless) x (--privileged, no --privileged) x (rootful, rootless) and the deeper I dig the more I'm confused.

So far, my understanding is that, as for the general principles:

• obviously rootless always uses a usernamespace
• in either mode (rootful or rootless) podman tries to use native overlayfs mount (I'm not sure if it falls back to fuse-overlayfs)
• in order to do this (or mount anything) it must have cap_sys_admin which it drops after the mount except when --privileged is used
• a rootful container, in general, does not use a usernamepace, so a sub-container would have to have its "host" container ran with --privileged in order to keep this cap whether a rootless one would be able to fresh start with all (namespaced) caps

I can see that the above podman image comes with a podman user with the following subuids (same for subgids) :

# cat /etc/subuid
podman:1:999
podman:1001:64535

I guess the first range is to be able to map to those uids which exists in the host container. I'm not really sure what the second one is intended for (maybe just have more uids) ?

I gave my own user the following subuids/subgids:

hummel:100000:65536

So, here's what I tested:

1. rootful from global usernamespace : no usernamespace created, with or without --privileged (seems normal to me)

2. rootful container from inside rootful container
   a) without --privileged : it seems it can neither use native mount (which is expected as stated above) but nor can it use fuse-overlayfs : is this because this requires a kernel module and only --privileged allows a container to use a module ?
   b) with --privileged: this works, no usernamespace created (as expected)

3. rootful container from inside rootless container
   a) without --privileged:
   I don't get this one : it seems podman wants to create another usernamespace and map the root user ? (which of course has no subuids)

 # podman run -it --rm quay.io/podman/stable /bin/bash
WARN[0000] Using rootless single mapping into the namespace. This might break some images.
Check /etc/subuid and /etc/subgid for adding sub*ids if not using a network user

-> what is happening here ?
b) with --privileged: it works and the rootful container stays in the same usernamespace as the one created by its rootless host (seems expected to me)

4. rootless container from global usernamespace
   Nothing special here

5. rootless container from rootful container
   I'm issuing from inside the rootless container su - podman:

[podman@e0615ccbcaf2 ~]$ podman run -it --rm quay.io/podman/stable /bin/bash
WARN[0000] "/" is not a shared mount, this could cause issues or missing mounts with rootle
ss containers

I'm not sure about the above warning (I guess it is because a new mnt namespace is created ?)
But I don't get what I see then (which happens in rootless from rootless too):

root@273a250a1e6d /]# echo $$
1
[root@273a250a1e6d /]# cat /proc/$$/uid_map
         0 4294967295 4294967295
[root@273a250a1e6d /]# cat /proc/self/uid_map
         0       1000          1
         1          1        999
      1000       1001      64535
[root@273a250a1e6d /]# ls -l /proc/self/ns/user
lrwxrwxrwx 1 root root 0 Oct 24 17:43 /proc/self/ns/user -> 'user:[4026534138]'
[root@273a250a1e6d /]# ls -l /proc/$$/ns/user
ls: cannot read symbolic link '/proc/1/ns/user': Permission denied
lrwxrwxrwx 1 nobody nobody 0 Oct 24 17:43 /proc/1/ns/user

Why is the shell not mapped but cat is ?
Why ls can read its usernamespace (which I can confirm is different from the global one) and not the shell ?

I get similar results with --privileged as well

Finally, I've read that for rootless subcontainer (using this image) I should run podman run --user podman, but I don't get the idea as my understanding is this instruct to run the container cmd/process with the internal podman user but what I want for running the rootless inner container is the external podman user (the one inside the outer image, which will maps the inner uids to is outer subuids) ? Or maybe it was meant to act as my su - podman from the outside ?

Can you help me figuring out what's going on and/or what I should or shouldn't do to run in those different scenarii podmain inside podman ?

Thanks for your help

--
Thomas HUMMEL

[rhatdan]

Have you read https://www.redhat.com/en/blog/podman-inside-container?

[th-hummel]

Hello, thanks for your reply

I did read it now.

I also was in the search to have something more restricted than --privileged which seems often too powerful in my view.

I indeed was somehow missing the --device and the unmask points and I did not notice that it was the image setup which instructed to use fuse-overlay instead of native overlay

In my understanding running su - podman should have the same result as initially running with --user podman (the basic idea being to be able to usernamespace with enough uids/gids)

I'm still confused about the results I got (especially the uid_maps mappings) described in my initial post.

Those unanswered questions made me in the meantime, as I reply privately to you, read more about the --userns option, leading to more questions I restate here :

For instance, using keep-id mode I can see the following mapping:

$ podman run -it --rm --userns=keep-id ubi8:latest /bin/bash
$ cat /proc/$$/uid_map
         0          1       1000
      1000          0          1
      1001       1001      64536

which actually accounts for the same number of subuids my user have

$ cat /etc/subuid
hummel:100000:65536

but my understanding was that the only way to have 0 on the second line would be to have a second usernamespace created by root from the one we would land on without the --userns option (or with host value) but I see only one CLONE_NEWUSER clone with strace.

Or in the auto mode, I don't understand what allows my rootless user to map (even in the hypothesis of a sub-subnamespace) to the subuid 1 which neither I (hummel) nor root "owns"

$ podman run -it --rm --userns=auto ubi8:latest /bin/bash
[root@dc09861a7bb8 /]# cat /proc/$$/uid_map
         0          1       1024

Thanks for your help

--
Thomas HUMMEL

[th-hummel]

So, basically for the --userns question above, the only way I can imagine it work is to nest 2 usernamespaces and since we are (mapped root) in the outer one, we can assign subuids/subgids to this root, then newuidmap what I see in the inner one.

About my initial post, I really cannot imagine in what situation I could have a different result between /proc/$$ and /proc/self regarding uid_maps. Or could it be because of some /proc/ masking ? (I don't see those /proc` dir beeing masked though)

I could be totally wrong with those 2 hypothesis.

--
Thomas HUMMEL