Quadlet containers in Quadlet pods ignore user namespace

[Syquel]

Issue Description

If a pod creates a new user namespace and containers are started with the option --pod-id-file <path> to associate the container with the pod, the container does not inherit the user namespace of the pod.
If the container is instead started with the option --pod <pod-name> the container successfully joins the user namespace of the pod.

This behavior can be observed in Quadlet containers which are associated with Quadlet pods, because those containers are started with the --pod-id-file <path> option.

Steps to reproduce the issue

Steps to reproduce the issue

[root@host]# podman pod create --pod-id-file=/run/userns-pod.pod-id --name=userns-pod --userns=auto  
12fb65c712557159938a2dd929fdb303a266c6b2809c5a98ed913382e95aab88  
[root@host]# cat /run/userns-pod.pod-id  
12fb65c712557159938a2dd929fdb303a266c6b2809c5a98ed913382e95aab88  
[root@host]# podman run --rm --pod-id-file /run/userns-pod.pod-id fedora:40 cat /proc/self/uid_map /proc/self/gid_map  
         0          0 4294967295  
         0          0 4294967295  
[root@host]# podman run --rm --pod userns-pod fedora:40 cat /proc/self/uid_map /proc/self/gid_map  
         0     173728       1024  
         0     173728       1024  

Describe the results you received

Containers started with the --pod-id-file option do not join the user namespace of the pod.

[root@host]# podman run --rm --pod-id-file /run/userns-pod.pod-id fedora:40 cat /proc/self/uid_map /proc/self/gid_map
         0          0 4294967295
         0          0 4294967295

Describe the results you expected

Containers started with the --pod-id-file option should behave the same way as containers started with the --pod option and join the user namespace of the pod.

[root@host]# podman run --rm --pod userns-pod fedora:40 cat /proc/self/uid_map /proc/self/gid_map
         0     173728       1024
         0     173728       1024

podman info output

host:
  arch: amd64
  buildahVersion: 1.36.0
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.10-1.fc40.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: '
  cpuUtilization:
    idlePercent: 97.66
    systemPercent: 1.48
    userPercent: 0.86
  cpus: 4
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    version: "40"
  eventLogger: journald
  freeLocks: 2013
  hostname: host
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.8.11-300.fc40.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 4913332224
  memTotal: 16531955712
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.10.0-1.fc40.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.10.0
    package: netavark-1.10.3-3.fc40.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.10.3
  ociRuntime:
    name: crun
    package: crun-1.15-1.fc40.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.15
      commit: e6eacaf4034e84185fd8780ac9262bbf57082278
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240510.g7288448-1.fc40.x86_64
    version: |
      pasta 0^20240510.g7288448-1.fc40.x86_64-pasta
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 8588881920
  swapTotal: 8589930496
  uptime: 72h 20m 25.00s (Approximately 3.00 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 11
    paused: 0
    running: 11
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.imagestore: /usr/lib/containers/storage
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 510337744896
  graphRootUsed: 42109374464
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 46
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 5.1.0
  Built: 1716940800
  BuiltTime: Wed May 29 02:00:00 2024
  GitCommit: ""
  GoVersion: go1.22.3
  Os: linux
  OsArch: linux/amd64
  Version: 5.1.0

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

Yes

Additional environment details

No response

Additional information

No response

[Syquel]

I ran one more test to observe the namespaces from the outside:
Preparation:

[root@host]# podman pod create --pod-id-file=/run/userns-pod.pod-id --name=userns-pod --userns=auto

Test with --pod-id-file:

[root@host]# podman run --rm --pod-id-file /run/userns-pod.pod-id --detach --name userns-container fedora:40 sleep 60
b7093dbc091e2089f43744dc8ea67dbce15fa7b1f6090ac09714c2abd96f8f30
[root@host]# podman pod top userns-pod user huser group hgroup pid hpid args
USER        HUSER       GROUP       HGROUP      PID         HPID        COMMAND
0           173728      0           173728      1           379882      /catatonit -P 
root        root        root        root        1           380712      sleep 60
[root@host]# podman container ps --ns
CONTAINER ID  NAMES                          PID         CGROUPNS    IPC         MNT         NET         PIDNS       USERNS      UTS
53f6fe75c6e9  0797a6f8a812-infra             379882      4026533299  4026533225  4026533223  4026533227  4026533226  4026533221  4026533224
b7093dbc091e  userns-container               380712      4026533304  4026533225  4026533302  4026533227  4026533303  4026531837  4026533224

Test with --pod:

[root@host]# podman run --rm --pod userns-pod --detach --name userns-container fedora:40 sleep 60
cf908a1e5c264764bb0c8b3df91741fddbaae3a87eef64bd821edb1ebe0c2032
[root@host]# podman pod top userns-pod user huser group hgroup pid hpid args
USER        HUSER       GROUP       HGROUP      PID         HPID        COMMAND
0           173728      0           173728      1           379882      /catatonit -P 
root        173728      root        173728      1           380979      sleep 60
[root@host]# podman container ps --ns
CONTAINER ID  NAMES                          PID         CGROUPNS    IPC         MNT         NET         PIDNS       USERNS      UTS
53f6fe75c6e9  0797a6f8a812-infra             379882      4026533299  4026533225  4026533223  4026533227  4026533226  4026533221  4026533224
cf908a1e5c26  userns-container               380979      4026533305  4026533225  4026533302  4026533227  4026533304  4026533221  4026533224

If --pod is used the user namespaces of the infra container and our container are the same while they are different if --pod-id-file is used.

And one more test to make sure this behavior is not related to the pod ID being used instead of the pod name:

[root@host]# podman pod create --name=userns-pod --userns=auto
76ec544f3888541c1ff15a19960b5628fe5b587233a3182ad2ea19f535890084
[root@host]# podman run --rm --pod 76ec544f3888541c1ff15a19960b5628fe5b587233a3182ad2ea19f535890084 --detach --name userns-container fedora:40 sleep 60
7827d74732b96c6de0191d63ad316ce82715ae39240a2f7984926e61f78fff57
[root@host]# podman pod top userns-pod user huser group hgroup pid hpid args
USER        HUSER       GROUP       HGROUP      PID         HPID        COMMAND
root        171680      root        171680      1           381781      sleep 60 
0           171680      0           171680      1           381695      /catatonit -P
[root@host]# podman container ps --ns
CONTAINER ID  NAMES                          PID         CGROUPNS    IPC         MNT         NET         PIDNS       USERNS      UTS
cd0fab6e41da  76ec544f3888-infra             381695      4026533193  4026533119  4026533117  4026533121  4026533120  4026532718  4026533118
7827d74732b9  userns-container               381781      4026533220  4026533119  4026533214  4026533121  4026533219  4026532718  4026533118

[Luap99]

Do you have the PODMAN_USERNS env set or userns in containers.conf. I see a problem in the code but it should not happen unless either one of these is set.

[Luap99]

Ok so it is also broken without having them set. It is not exactly clear to me why but #23082 fixes it so feel free to test that.