Podman errors on tar with potentially insufficient UIDs or GIDs available in user namespace

[AClon314]

Getting image source signatures
Copying blob c6a83fedfae6 done   | 
Error: copying system image from manifest list: writing blob: adding layer with blob "sha256:c6a83fedfae6ed8a4f5f7cbb6a7b6f1c1ec3d86fea8cb9e5ba2e5e6673fde9f6":
    processing tar file(potentially insufficient UIDs or GIDs available in user namespace (requested 0:42 for /etc/shadow):
        Check /etc/subuid and /etc/subgid if configured locally and run "podman system migrate": lchown /etc/shadow: invalid argument): exit status 1

podman info

host:
  arch: amd64
  buildahVersion: 1.33.7
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon_2.1.10+ds1-1build2_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: unknown'
  cpuUtilization:
    idlePercent: 96.15
    systemPercent: 1.81
    userPercent: 2.04
  cpus: 12
  databaseBackend: sqlite
  distribution:
    codename: wilma
    distribution: linuxmint
    version: "22"
  eventLogger: journald
  freeLocks: 2048
  hostname: r7000
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
  kernel: 6.8.0-41-generic
  linkmode: dynamic
  logDriver: journald
  memFree: 10960297984
  memTotal: 20813598720
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns_1.4.0-5_amd64
      path: /usr/lib/podman/aardvark-dns
      version: aardvark-dns 1.4.0
    package: netavark_1.4.0-4_amd64
    path: /usr/lib/podman/netavark
    version: netavark 1.4.0
  ociRuntime:
    name: crun
    package: crun_1.14.1-1_amd64
    path: /usr/bin/crun
    version: |-
      crun version 1.14.1
      commit: de537a7965bfbe9992e2cfae0baeb56a08128171
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt_0.0~git20240220.1e6f92b-1_amd64
    version: |
      pasta unknown version
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.2.1-1build2_amd64
    version: |-
      slirp4netns version 1.2.1
      commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.5
  swapFree: 0
  swapTotal: 0
  uptime: 1h 6m 55.00s (Approximately 0.04 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/n/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/n/.local/share/containers/storage
  graphRootAllocated: 421606629376
  graphRootUsed: 148685017088
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/n/.local/share/containers/storage/volumes
version:
  APIVersion: 4.9.3
  Built: 0
  BuiltTime: Thu Jan  1 08:00:00 1970
  GitCommit: ""
  GoVersion: go1.22.2
  Os: linux
  OsArch: linux/amd64
  Version: 4.9.3

podman run log

❯ podman --log-level debug run --rm alpine echo "Hello, Podman"

INFO[0000] podman filtering at log level debug          
DEBU[0000] Called run.PersistentPreRunE(podman --log-level debug run --rm alpine echo Hello, Podman) 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
INFO[0000] Using sqlite as database backend             
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/n/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /home/n/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /home/n/.local/share/containers/storage/volumes 
DEBU[0000] Using transient store: false                 
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that metacopy is not being used 
DEBU[0000] Cached value indicated that native-diff is usable 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument 
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument 
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument 
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument 
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument 
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument 
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
DEBU[0000] Successfully loaded network podman: &{podman 2f259bab93aaaaa2542ba43ef33eb990d0999ee1b9924b557b7be53c0b7a1bb9 bridge cni-podman0 2024-08-08 11:31:05.975452042 +0800 CST [{{{10.88.0.0 ffff0000}} 10.88.0.1 <nil>}] [] false false false [] map[] map[] map[driver:host-local]} 
DEBU[0000] Successfully loaded 1 networks               
INFO[0000] Setting parallel job count to 37             
DEBU[0000] Successfully loaded 1 networks               
DEBU[0000] Pulling image alpine (policy: missing)       
DEBU[0000] Looking up image "alpine" in local containers storage 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/shortnames.conf" 
DEBU[0000] Trying "docker.io/library/alpine:latest" ... 
DEBU[0000] reference "[overlay@/home/n/.local/share/containers/storage+/run/user/1000/containers]docker.io/library/alpine:latest" does not resolve to an image ID 
DEBU[0000] Trying "localhost/alpine:latest" ...         
DEBU[0000] reference "[overlay@/home/n/.local/share/containers/storage+/run/user/1000/containers]localhost/alpine:latest" does not resolve to an image ID 
DEBU[0000] Trying "docker.io/library/alpine:latest" ... 
DEBU[0000] reference "[overlay@/home/n/.local/share/containers/storage+/run/user/1000/containers]docker.io/library/alpine:latest" does not resolve to an image ID 
DEBU[0000] Trying "alpine" ...                          
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Attempting to pull candidate docker.io/library/alpine:latest for alpine 
DEBU[0000] parsed reference into "[overlay@/home/n/.local/share/containers/storage+/run/user/1000/containers]docker.io/library/alpine:latest" 
DEBU[0000] Resolved "alpine" as an alias (/etc/containers/registries.conf.d/shortnames.conf) 
Resolved "alpine" as an alias (/etc/containers/registries.conf.d/shortnames.conf)
Trying to pull docker.io/library/alpine:latest...
DEBU[0000] Copying source image //alpine:latest to destination image [overlay@/home/n/.local/share/containers/storage+/run/user/1000/containers]docker.io/library/alpine:latest 
DEBU[0000] Using registries.d directory /etc/containers/registries.d 
DEBU[0000] Trying to access "docker.io/library/alpine:latest" 
DEBU[0000] No credentials matching docker.io/library/alpine found in /run/user/1000/containers/auth.json 
DEBU[0000] No credentials matching docker.io/library/alpine found in /home/n/.config/containers/auth.json 
DEBU[0000] No credentials matching docker.io/library/alpine found in /home/n/.docker/config.json 
DEBU[0000] No credentials matching docker.io/library/alpine found in /home/n/.dockercfg 
DEBU[0000] No credentials for docker.io/library/alpine found 
DEBU[0000]  No signature storage configuration found for docker.io/library/alpine:latest, using built-in default file:///home/n/.local/share/containers/sigstore 
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/docker.io 
DEBU[0000] GET https://registry-1.docker.io/v2/         
DEBU[0002] Ping https://registry-1.docker.io/v2/ status 401 
DEBU[0002] GET https://auth.docker.io/token?scope=repository%3Alibrary%2Falpine%3Apull&service=registry.docker.io 
DEBU[0004] GET https://registry-1.docker.io/v2/library/alpine/manifests/latest 
DEBU[0006] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.list.v2+json" 
DEBU[0006] Using SQLite blob info cache at /home/n/.local/share/containers/cache/blob-info-cache-v1.sqlite 
DEBU[0006] Source is a manifest list; copying (only) instance sha256:eddacbc7e24bf8799a4ed3cdcfa50d4b88a323695ad80f317b6629883b2c2a78 for current system 
DEBU[0006] GET https://registry-1.docker.io/v2/library/alpine/manifests/sha256:eddacbc7e24bf8799a4ed3cdcfa50d4b88a323695ad80f317b6629883b2c2a78 
DEBU[0007] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.v2+json" 
DEBU[0007] IsRunningImageAllowed for image docker:docker.io/library/alpine:latest 
DEBU[0007]  Using default policy section                
DEBU[0007]  Requirement 0: allowed                      
DEBU[0007] Overall: allowed                             
DEBU[0007] Downloading /v2/library/alpine/blobs/sha256:324bc02ae1231fd9255658c128086395d3fa0aedd5a41ab6b034fd649d1a9260 
DEBU[0007] GET https://registry-1.docker.io/v2/library/alpine/blobs/sha256:324bc02ae1231fd9255658c128086395d3fa0aedd5a41ab6b034fd649d1a9260 
Getting image source signatures
DEBU[0008] Reading /home/n/.local/share/containers/sigstore/library/alpine@sha256=eddacbc7e24bf8799a4ed3cdcfa50d4b88a323695ad80f317b6629883b2c2a78/signature-1 
DEBU[0008] Not looking for sigstore attachments: disabled by configuration 
DEBU[0008] Manifest has MIME type application/vnd.docker.distribution.manifest.v2+json, ordered candidate list [application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v1+json] 
DEBU[0008] ... will first try using the original manifest unmodified 
DEBU[0008] Checking if we can reuse blob sha256:c6a83fedfae6ed8a4f5f7cbb6a7b6f1c1ec3d86fea8cb9e5ba2e5e6673fde9f6: general substitution = true, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true 
DEBU[0008] Failed to retrieve partial blob: convert_images not configured 
DEBU[0008] Downloading /v2/library/alpine/blobs/sha256:c6a83fedfae6ed8a4f5f7cbb6a7b6f1c1ec3d86fea8cb9e5ba2e5e6673fde9f6 
DEBU[0008] GET https://registry-1.docker.io/v2/library/alpine/blobs/sha256:c6a83fedfae6ed8a4f5f7cbb6a7b6f1c1ec3d86fea8cb9e5ba2e5e6673fde9f6 
Copying blob c6a83fedfae6 [--------------------------------------] 0.0b / 3.5MiB | 0.0 b/s
DEBU[0009] Detected compression format gzip             
Copying blob c6a83fedfae6 done   | 
DEBU[0022] Cached value indicated that idmapped mounts for overlay are not supported 
DEBU[0022] Check for idmapped mounts support            
DEBU[0022] Applying tar in /home/n/.local/share/containers/storage/overlay/78561cef0761903dd2f7d09856150a6d4fb48967a8f113f3e33d79effbf59a07/diff 
DEBU[0022] Error pulling candidate docker.io/library/alpine:latest: copying system image from manifest list: writing blob: adding layer with blob "sha256:c6a83fedfae6ed8a4f5f7cbb6a7b6f1c1ec3d86fea8cb9e5ba2e5e6673fde9f6": processing tar file(potentially insufficient UIDs or GIDs available in user namespace (requested 0:42 for /etc/shadow): Check /etc/subuid and /etc/subgid if configured locally and run "podman system migrate": lchown /etc/shadow: invalid argument): exit status 1 
Error: copying system image from manifest list: writing blob: adding layer with blob "sha256:c6a83fedfae6ed8a4f5f7cbb6a7b6f1c1ec3d86fea8cb9e5ba2e5e6673fde9f6": processing tar file(potentially insufficient UIDs or GIDs available in user namespace (requested 0:42 for /etc/shadow): Check /etc/subuid and /etc/subgid if configured locally and run "podman system migrate": lchown /etc/shadow: invalid argument): exit status 1
DEBU[0022] Shutting down engines

---

❯ cat /etc/subuid /etc/subgid
root:100000:65536
# n:165536:65536

root:100000:65536
# n:165536:65536

❯ sudo getcap /bin/newgidmap /bin/newuidmap 
/bin/newgidmap cap_setgid=eip
/bin/newuidmap cap_setuid=eip

I've tried lots of solutions for subuid, subgid, but still not working. I've compiled podman v5.2.2 from source and get same result :(

OS: Linux mint 22 (Ubuntu 24.04 Noble)

[AClon314]

Temporary solution:

vim ~/.config/containers/storage.conf

Add these lines:

[storage.options.overlay]
ignore_chown_errors = "true"

Output:

❯ podman run --rm alpine echo "Hello, Podman"        

WARN[0000] Additional gid=1 is not present in the user namespace, skip setting it 
WARN[0000] Additional gid=2 is not present in the user namespace, skip setting it 
WARN[0000] Additional gid=3 is not present in the user namespace, skip setting it 
WARN[0000] Additional gid=4 is not present in the user namespace, skip setting it 
WARN[0000] Additional gid=6 is not present in the user namespace, skip setting it 
WARN[0000] Additional gid=10 is not present in the user namespace, skip setting it 
WARN[0000] Additional gid=11 is not present in the user namespace, skip setting it 
WARN[0000] Additional gid=20 is not present in the user namespace, skip setting it 
WARN[0000] Additional gid=26 is not present in the user namespace, skip setting it 
WARN[0000] Additional gid=27 is not present in the user namespace, skip setting it 
Hello, Podman

References:
https://unix.stackexchange.com/questions/689175/podman-errors-on-tar-with-potentially-insufficient-uids-or-gids-available-in-use
https://github.com/containers/podman/blob/main/vendor/github.com/containers/storage/storage.conf

[nalind]

The podman info output is confirming that there are no IDs available for the user to map to ID 42, and the configuration in /etc/subuid and /etc/subgid that would provide them are commented out. Uncomment them, use podman system reset to make sure podman picks up the changes, and try again?

[AClon314]

thanks, but finally I've solve that by turn -rws into -rwx, which removes the SUID/SGID limit to execute it!

[AClon314]

I've gather some possible solutions for subgid/subuid problems.

Ultimate solution:

range=100000-200000
sudo usermod --add-subuids $range --add-subgids $range $(whoami)

chmod 0755 /usr/bin/newuidmap /usr/bin/newgidmap

vim ~/.config/containers/storage.conf

Add these 2 lines in ~/.config/containers/storage.conf

[storage.options.overlay]
ignore_chown_errors = "true"

---

My problems are:

• Not set up /etc/subuid /etc/subgid
• Broken permissions for newuidmap newgidmap