--userns=keep-id freezes Podman with some Docker containers

[brndd]

Issue Description

Some Docker containers do complicated user management in entrypoint scripts after container creation, running initially as root and using a shell script to create the actual user the container is run as. When paired with --userns=keep-id, this can cause Podman to become unresponsive (podman ps, podman stop, and other commands stop working entirely) until the podman process that launched the container is killed manually.

I have not produced a minimal reproducible example, but one affected container is the ShokoServer container (https://hub.docker.com/r/shokoanime/server), which has an entrypoint script that looks like this: https://github.com/ShokoAnime/ShokoServer/blob/master/dockerentry.sh

Steps to reproduce the issue

1. Attempt to run the shokoserver container as thus: podman run --name myshoko --restart always --userns=keep-id shokoanime/server:latest
2. The container will complain about the entrypoint's user commands failing
3. Podman will hang seemingly indefinitely

Describe the results you received

Podman hangs seemingly indefinitely. Ctrl-C does not manage to exit the session. podman ps, podman stop myshoko, and other podman commands (in other shells) will also hang. The only way to un-hang podman I've found is to kill the affected podman process from another shell.

Describe the results you expected

Podman doesn't hang. Ideally the container would also work, but the user management stuff it's doing seems to be a pretty Docker-specific hack so reconciling it with Podman and --userns=keep-id may not be easy.

podman info output

host:
  arch: amd64
  buildahVersion: 1.33.5
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.10-1.fc39.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: '
  cpuUtilization:
    idlePercent: 98.35
    systemPercent: 0.57
    userPercent: 1.08
  cpus: 16
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: server
    version: "39"
  eventLogger: journald
  freeLocks: 2027
  hostname: junker
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.7.7-200.fc39.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 569647104
  memTotal: 33551212544
  networkBackend: cni
  networkBackendInfo:
    backend: cni
    dns:
      package: podman-plugins-4.9.3-1.fc39.x86_64
      path: /usr/libexec/cni/dnsname
      version: |-
        CNI dnsname plugin
        version: 1.3.1
        commit: unknown
    package: |-
      containernetworking-plugins-1.3.0-3.fc39.x86_64
      podman-plugins-4.9.3-1.fc39.x86_64
    path: /usr/libexec/cni
  ociRuntime:
    name: crun
    package: crun-1.14.3-1.fc39.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.14.3
      commit: 1961d211ba98f532ea52d2e80f4c20359f241a98
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240220.g1e6f92b-1.fc39.x86_64
    version: |
      pasta 0^20240220.g1e6f92b-1.fc39.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-1.fc39.x86_64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 17176190976
  swapTotal: 17179860992
  uptime: 1h 26m 24.00s (Approximately 0.04 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/eru/.config/containers/storage.conf
  containerStore:
    number: 7
    paused: 0
    running: 2
    stopped: 5
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/eru/.local/share/containers/storage
  graphRootAllocated: 490940137472
  graphRootUsed: 76374577152
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 146
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/eru/.local/share/containers/storage/volumes
version:
  APIVersion: 4.9.3
  Built: 1708357294
  BuiltTime: Mon Feb 19 17:41:34 2024
  GitCommit: ""
  GoVersion: go1.21.7
  Os: linux
  OsArch: linux/amd64
  Version: 4.9.3

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[baude]

it is the container cleanup @mheon

[mheon]

The --userns=keep-id command is also setting user to non-root - so I am wondering if these init scripts that are written assuming root can ever work properly in that environment?
That aside, this is definitely a bug.

[baude]

documenting what i see, idont have time to run with this more rn. running the reproducer provider, the does occur as described. The terminal is returned. but indeed, a lock is being taken (or never freed). I then observe one conmon and one podman process continuing, Both processes are sitting in a wait (podman in a futex).

[Luap99]

@baude Are you running on main? Did you try without --restart always?
This sounds like the same as #21477 which was fixed in #21522

[brndd]

I can confirm it doesn't occur without --restart always on my end.

[baude]

@baude Are you running on main? Did you try without --restart always? This sounds like the same as #21477 which was fixed in #21522

I was not using main ...

[Luap99]

Closing as dup then, the fix will be in podman 5.0
Duplicate of #21477