call to podman system migrate fails on newgidmap if user id is high

[wucke13]

Issue Description

I want to run podman in rootless mode. Due to being an AD user, I have quite a big UID, somewhere around 130 000. I have the following subuid/subgid files:

username:600001:65536

Now when I run podman system migrate, I get the following error (let's assume my UID is 138000):

ERRO[0000] running `/run/wrappers/bin/newuidmap 33632 0 138000 1 1 600001 65536`:  
Error: cannot set up namespace using "/run/wrappers/bin/newuidmap": exit status 1

If I increase the subuid/subgid subordinate size to bee bigger than my uid, than everything works:

username:600001:655360

Steps to reproduce the issue

Steps to reproduce the issue

1. Create user with a big uid (bigger than the quasi-default 2^16 / 65536)
2. Make sure the user has subuid/subgid subordinates smaller or equal to 2^16
3. Run podman system migrate

Describe the results you received

ERRO[0000] running `/run/wrappers/bin/newuidmap 33632 0 138000 1 1 600001 65536`:  
Error: cannot set up namespace using "/run/wrappers/bin/newuidmap": exit status 1

Describe the results you expected

podman works just fine, because irrespective of my real user id being quite big, I don't use any images that use more than 16^2 uids

podman info output

host:
  arch: amd64
  buildahVersion: 1.32.0
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: Unknown
    path: /run/current-system/sw/bin/conmon
    version: 'conmon version 2.1.8, commit: '
  cpuUtilization:
    idlePercent: 96.23
    systemPercent: 0.98
    userPercent: 2.79
  cpus: 12
  databaseBackend: boltdb
  distribution:
    codename: tapir
    distribution: nixos
    version: "23.11"
  eventLogger: journald
  freeLocks: 2045
  hostname: mymachine
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 433468
      size: 1
    - container_id: 1
      host_id: 600001
      size: 655360
    uidmap:
    - container_id: 0
      host_id: 138000
      size: 1
    - container_id: 1
      host_id: 600001
      size: 655360
  kernel: 6.6.3
  linkmode: dynamic
  logDriver: journald
  memFree: 20075917312
  memTotal: 33113714688
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: Unknown
      path: /nix/store/nm14aplcsxyjx0rl6l46rp5rv8q2jghq-podman-4.7.2/libexec/podman/aardvark-dns
      version: aardvark-dns 1.8.0
    package: Unknown
    path: /nix/store/nm14aplcsxyjx0rl6l46rp5rv8q2jghq-podman-4.7.2/libexec/podman/netavark
    version: netavark 1.7.0
  ociRuntime:
    name: crun
    package: Unknown
    path: /nix/store/6b220wi4jj6025iqyhsh3sg0r64yhbjf-crun-1.11.1/bin/crun
    version: |-
      crun version 1.11.1
      commit: 1.11.1
      rundir: /run/user/138000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    exists: false
    path: /run/user/138000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: ""
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /nix/store/nm14aplcsxyjx0rl6l46rp5rv8q2jghq-podman-4.7.2/libexec/podman/slirp4netns
    package: Unknown
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 0
  swapTotal: 0
  uptime: 2h 57m 47.00s (Approximately 0.08 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /home/username/.config/containers/storage.conf
  containerStore:
    number: 3
    paused: 0
    running: 0
    stopped: 3
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/username/.local/share/containers/storage
  graphRootAllocated: 1023667408896
  graphRootUsed: 742521540608
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 5
  runRoot: /run/user/138000/containers
  transientStore: false
  volumePath: /home/username/.local/share/containers/storage/volumes
version:
  APIVersion: 4.7.2
  Built: 315532800
  BuiltTime: Tue Jan  1 01:00:00 1980
  GitCommit: ""
  GoVersion: go1.21.4
  Os: linux
  OsArch: linux/amd64
  Version: 4.7.2

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

My user account is provided using sssd with an AD backend

Additional information

No response

[wucke13]

This bug report, my observations where based on wrong assumptions.

More detail: I used the NixOS environment.etc.<name>.text option to create the subuid/subgid files on my machine. Why files? Because actually these are symlinks:

ls /etc/subuid -l
lrwxrwxrwx - root 15 Dez 17:49 /etc/subuid -> /etc/static/subuid
ls /etc/static/subuid -l
lrwxrwxrwx - root  1 Jan  1970 /etc/static/subuid -> /nix/store/bi1f79pc2cpa70v7hi7xmd65r1n4918h-etc-subuid

So, this are not files, but double-indirection symlink pairs pointing to read only files in the nix store, which is mounted ro. In order to quickly try different settings in the subuid file, I'd remove the original symlink and instead put a file there, that I can edit. Seems like almost the same, having a file or a symlink to a file. But, newuidmap opens the file with O_NOFOLLOW.

O_NOFOLLOW: If the trailing component (i.e., basename) of pathname is a symbolic link, then the open fails, with the error ELOOP.

So, once I made my small test changes, I unbeknownst to me also changed something that decided whether newuidmap would succeed or fail in opening the subuid/subgid file. Here are the last lines of a strace from a failing newuidmap call:

openat(AT_FDCWD, "/etc/subuid", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_NOFOLLOW|O_CLOEXEC) = -1 ELOOP (Too many levels of symbolic links)
exit_group(1)                           = ?
+++ exited with 1 +++

A simple fix on NixOS is the usage of the environment.etc.<name>.mode option, to force creation of a file (instead of symlink-mania).

So, next time when some users complain about newuidmap just failing, ask them as if the subuid and subgid files are actually files, and not symlinks...