Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

"lstat: invalid argument" when using 'overlay' storage driver with fuse-overlayfs #9006

Closed
Ristovski opened this issue Jan 18, 2021 · 15 comments
Closed

"lstat: invalid argument" when using 'overlay' storage driver with fuse-overlayfs #9006

Ristovski opened this issue Jan 18, 2021 · 15 comments
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.

Comments

@Ristovski
Copy link

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Running podman in rootless mode with the overlay (or overlay2, they are the same from what I can tell) storage driver (with fuse-overlayfs) fails.

Steps to reproduce the issue:

  1. Run podman container run --rm -it hello-world

Describe the results you received:

Running anything with overlayfs on Gentoo causes the following error:

Error: error resolving storage path for container [id]: lstat /home/[user]/.local/share/containers/storage/overlay2/[id]/merged: invalid argument

I modified container_internal.go to exit as soon as it hits that error (to prevent any cleanup),
and the container [id] directory ends up containing only the following items:

$ find .
./merged
./work
./work/work
./lower
./link
./diff

in storage driver directory:
(note the hello binary being present)

rafael@RPC ~/.local/share/containers/storage/overlay2 $ find .
.
./9c27e219663c25e0f28493790cc0b88bc973ba3b1686355f221c38a36978ac63
./9c27e219663c25e0f28493790cc0b88bc973ba3b1686355f221c38a36978ac63/empty
./9c27e219663c25e0f28493790cc0b88bc973ba3b1686355f221c38a36978ac63/merged
./9c27e219663c25e0f28493790cc0b88bc973ba3b1686355f221c38a36978ac63/work
./9c27e219663c25e0f28493790cc0b88bc973ba3b1686355f221c38a36978ac63/link
./9c27e219663c25e0f28493790cc0b88bc973ba3b1686355f221c38a36978ac63/diff
./9c27e219663c25e0f28493790cc0b88bc973ba3b1686355f221c38a36978ac63/diff/hello
./l
./l/F7MDBH2YNJJOT3IHFYIKL3G4OW
./l/AFHTK4I5REAE6ZYPU6QKDFHYZO
./37fd79dd6c7c9919b12660c74ffe4312fd36b5bda10448f1f1d3a5535504a021
./37fd79dd6c7c9919b12660c74ffe4312fd36b5bda10448f1f1d3a5535504a021/merged
./37fd79dd6c7c9919b12660c74ffe4312fd36b5bda10448f1f1d3a5535504a021/work
./37fd79dd6c7c9919b12660c74ffe4312fd36b5bda10448f1f1d3a5535504a021/work/work
./37fd79dd6c7c9919b12660c74ffe4312fd36b5bda10448f1f1d3a5535504a021/lower
./37fd79dd6c7c9919b12660c74ffe4312fd36b5bda10448f1f1d3a5535504a021/link
./37fd79dd6c7c9919b12660c74ffe4312fd36b5bda10448f1f1d3a5535504a021/diff

Additional information you deem important (e.g. issue happens only occasionally):

vfs storage driver works just fine

Output of podman version:

podman version 2.2.1

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.18.0
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: Unknown
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.22, commit: 9c34a8663b85e479e0c083801e89a2b2835228ed'
  cpus: 4
  distribution:
    distribution: gentoo
    version: unknown
  eventLogger: file
  hostname: RPC
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 1065536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 1065536
      size: 65536
  kernel: 5.5.0RMOD-00003-g35c07c7e8a29
  linkmode: dynamic
  memFree: 786419712
  memTotal: 8217759744
  ociRuntime:
    name: crun
    package: Unknown
    path: /usr/bin/crun
    version: |-
      crun version 0.15
      commit: 56ca95e61639510c7dbd39ff512f80f626404969
      spec: 1.0.0
      +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: Unknown
    version: |-
      slirp4netns version 1.1.8
      commit: d361001f495417b880f20329121e3aa431a8f90f
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.3.0
  swapFree: 0
  swapTotal: 0
  uptime: 117h 4m 17.82s (Approximately 4.88 days)
registries:
  docker.io:
    Blocked: false
    Insecure: false
    Location: mirror.gcr.io
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: docker.io
  search:
  - docker.io
store:
  configFile: /home/rafael/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay2.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: Unknown
      Version: |-
        fusermount3 version: 3.9.1
        fuse-overlayfs: version 1.3
        FUSE library version 3.9.1
        using FUSE kernel interface version 7.31
  graphRoot: /home/rafael/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 1
  runRoot: /tmp/containers-user-1000/containers
  volumePath: /home/rafael/.local/share/containers/storage/volumes
version:
  APIVersion: 2.1.0
  Built: 1610749743
  BuiltTime: Fri Jan 15 23:29:03 2021
  GitCommit: d8868746fd3f45b9bfd79e37bf2aa0c7103245fd
  GoVersion: go1.15.6
  OsArch: linux/amd64
  Version: 2.2.1

Package info (e.g. output of rpm -q podman or apt list podman):

app-emulation/libpod-2.2.1

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

Physical Gentoo Linux

storage.conf:

[storage]
  driver = "overlay2"
  graphroot = "/home/rafael/.local/share/containers/storage"
[storage.options]
mount_program = "/usr/bin/fuse-overlayfs"
additionalimagestores = [
]
@openshift-ci-robot openshift-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Jan 18, 2021
@Ristovski Ristovski changed the title lstat: invalid argument when using 'overlay' storage driver "lstat: invalid argument" when using 'overlay' storage driver Jan 18, 2021
@Ristovski
Copy link
Author

Ristovski commented Jan 18, 2021 •
edited
Loading

Edit: This does not end up using fuse-overlayfs.

So that was weird. I tried removing the storage.conf, and it started working. Perplexed by this, I decided to diff the output of podman info and got the following:

<   graphOptions:
<     overlay.mount_program:
<       Executable: /usr/bin/fuse-overlayfs
<       Package: Unknown
<       Version: |-
<         fusermount3 version: 3.9.1
<         fuse-overlayfs: version 1.3
<         FUSE library version 3.9.1
<         using FUSE kernel interface version 7.31

It appears that the

[storage.options]
mount_program = "/usr/bin/fuse-overlayfs"

entry in the storage.conf was causing this.

The working podman info reports graphOptions = {}.

Do note the mount_program entry is mentioned in https://github.com/containers/podman/blob/master/docs/tutorials/rootless_tutorial.md

If Podman is used before fuse-overlayfs is installed, it may be necessary to adjust the storage.conf file (see "User Configuration Files" below) to change the driver option under [storage] to "overlay" and point the mount_program option in [storage.options] to the path of the fuse-overlayfs executable

@afbjorklund
Copy link
Contributor

afbjorklund commented Jan 18, 2021 •
edited
Loading

The storage options got moved to specific drivers, here:
containers/storage@ff125a5

[storage.options.overlay]
mount_program = "/usr/bin/fuse-overlayfs"

BTW; I think "overlay2" is for docker, it got called "overlay" in podman

@Ristovski
Copy link
Author

Reopening this, since running the overlay storage driver without fuse-overlayfs working is a sub-optimal solution.

@Ristovski Ristovski reopened this Jan 18, 2021
@Ristovski Ristovski changed the title "lstat: invalid argument" when using 'overlay' storage driver "lstat: invalid argument" when using 'overlay' storage driver with fuse-overlayfs Jan 18, 2021
@Ristovski
Copy link
Author

I compared the output of a working install on a Ubuntu server and there seems to be something weird going on:

Working log:

[...]
DEBU[0001] Handling terminal attach                     
DEBU[0001] overlay: mount_data=nodev,lowerdir=/home/[user]/.local/share/containers/storage/overlay/l/QZK56ZJMVH4IRKMOBVNRBX465A,upperdir=/home/[user]/.local/share/containers/storage/overlay/6b7ec746d59a955f1c38976b88cd5ae715fce5470d5513dff667ac0c99c6b218/diff,workdir=/home/[user]/.local/share/containers/storage/overlay/6b7ec746d59a955f1c38976b88cd5ae715fce5470d5513dff667ac0c99c6b218/work 
DEBU[0001] Made network namespace at /run/user/1015/netns/cni-882f498f-8d55-b433-9911-f5b2088554ed for container 7d6030b9b2fdb857aa49e3311ddbf7cf07c89480b10f164a9a7910ba2860bedb 
DEBU[0001] mounted container "7d6030b9b2fdb857aa49e3311ddbf7cf07c89480b10f164a9a7910ba2860bedb" at "/home/[user]/.local/share/containers/storage/overlay/6b7ec746d59a955f1c38976b88cd5ae715fce5470d5513dff667ac0c99c6b218/merged" 
DEBU[0001] slirp4netns command: /usr/bin/slirp4netns --disable-host-loopback --mtu 65520 --enable-sandbox --enable-seccomp -c -e 3 -r 4 --netns-type=path /run/user/1015/netns/cni-882f498f-8d55-b433-9911-f5b2088554ed tap0 
DEBU[0001] Created root filesystem for container 7d6030b9b2fdb857aa49e3311ddbf7cf07c89480b10f164a9a7910ba2860bedb at /home/[user]/.local/share/containers/storage/overlay/6b7ec746d59a955f1c38976b88cd5ae715fce5470d5513dff667ac0c99c6b218/merged 
DEBU[0001] /etc/system-fips does not exist on host, not mounting FIPS mode secret 
DEBU[0001] reading hooks from /usr/share/containers/oci/hooks.d 
DEBU[0001] Created OCI spec for container 7d6030b9b2fdb857aa49e3311ddbf7cf07c89480b10f164a9a7910ba2860bedb at /home/[user]/.local/share/containers/storage/overlay-containers/7d6030b9b2fdb857aa49e3311ddbf7cf07c89480b10f164a9a7910ba2860bedb/userdata/config.json 
DEBU[0001] /usr/libexec/podman/conmon messages will be logged to syslog 
DEBU[0001] running conmon: /usr/libexec/podman/conmon    args="[--api-version 1 -c 7d6030b9b2fdb857aa49e3311ddbf7cf07c89480b10f164a9a7910ba2860bedb -u 7d6030b9b2fdb857aa49e3311ddbf7cf07c89480b10f164a9a7910ba2860bedb -r /usr/lib/cri-o-runc/sbin/runc -b /home/[user]/.local/share/containers/storage/overlay-containers/7d6030b9b2fdb857aa49e3311ddbf7cf07c89480b10f164a9a7910ba2860bedb/userdata -p /run/user/1015/containers/overlay-containers/7d6030b9b2fdb857aa49e3311ddbf7cf07c89480b10f164a9a7910ba2860bedb/userdata/pidfile -n loving_bardeen --exit-dir /run/user/1015/libpod/tmp/exits --socket-dir-path /run/user/1015/libpod/tmp/socket -l k8s-file:/home/[user]/.local/share/containers/storage/overlay-containers/7d6030b9b2fdb857aa49e3311ddbf7cf07c89480b10f164a9a7910ba2860bedb/userdata/ctr.log --log-level debug --syslog -t --conmon-pidfile /run/user/1015/containers/overlay-containers/7d6030b9b2fdb857aa49e3311ddbf7cf07c89480b10f164a9a7910ba2860bedb/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /home/[user]/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1015/containers --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg cgroupfs --exit-command-arg --tmpdir --exit-command-arg /run/user/1015/libpod/tmp --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.ignore_chown_errors=true --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/usr/bin/fuse-overlayfs --exit-command-arg --storage-opt --exit-command-arg overlay.mountopt=nodev --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg --syslog --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg 7d6030b9b2fdb857aa49e3311ddbf7cf07c89480b10f164a9a7910ba2860bedb]"
WARN[0001] Failed to add conmon to cgroupfs sandbox cgroup: error creating cgroup for cpu: mkdir /sys/fs/cgroup/cpu/libpod_parent: permission denied 
DEBU[0001] Received: 2181470                            
INFO[0001] Got Conmon PID as 2181459                    
DEBU[0001] Created container 7d6030b9b2fdb857aa49e3311ddbf7cf07c89480b10f164a9a7910ba2860bedb in OCI runtime 
DEBU[0001] Attaching to container 7d6030b9b2fdb857aa49e3311ddbf7cf07c89480b10f164a9a7910ba2860bedb 
DEBU[0001] connecting to socket /run/user/1015/libpod/tmp/socket/7d6030b9b2fdb857aa49e3311ddbf7cf07c89480b10f164a9a7910ba2860bedb/attach 
DEBU[0001] Received a resize event: {Width:227 Height:62} 
DEBU[0001] Starting container 7d6030b9b2fdb857aa49e3311ddbf7cf07c89480b10f164a9a7910ba2860bedb with command [/hello] 
DEBU[0001] Started container 7d6030b9b2fdb857aa49e3311ddbf7cf07c89480b10f164a9a7910ba2860bedb 

[SNIPPED container output]

DEBU[0001] Enabling signal proxying                     
DEBU[0001] Checking if container 7d6030b9b2fdb857aa49e3311ddbf7cf07c89480b10f164a9a7910ba2860bedb should restart 
DEBU[0001] Removing container 7d6030b9b2fdb857aa49e3311ddbf7cf07c89480b10f164a9a7910ba2860bedb 
DEBU[0001] Removing all exec sessions for container 7d6030b9b2fdb857aa49e3311ddbf7cf07c89480b10f164a9a7910ba2860bedb 
DEBU[0002] Cleaning up container 7d6030b9b2fdb857aa49e3311ddbf7cf07c89480b10f164a9a7910ba2860bedb 
DEBU[0002] Tearing down network namespace at /run/user/1015/netns/cni-882f498f-8d55-b433-9911-f5b2088554ed for container 7d6030b9b2fdb857aa49e3311ddbf7cf07c89480b10f164a9a7910ba2860bedb 
DEBU[0002] Successfully cleaned up container 7d6030b9b2fdb857aa49e3311ddbf7cf07c89480b10f164a9a7910ba2860bedb 
DEBU[0002] unmounted container "7d6030b9b2fdb857aa49e3311ddbf7cf07c89480b10f164a9a7910ba2860bedb" 
DEBU[0002] Container 7d6030b9b2fdb857aa49e3311ddbf7cf07c89480b10f164a9a7910ba2860bedb storage is already unmounted, skipping... 
DEBU[0002] Called run.PersistentPostRunE(podman --log-level=debug container run --rm -it hello-world)
[...]

Broken log:

[...]
DEBU[0000] Handling terminal attach                     
DEBU[0000] overlay: mount_data=nodev,lowerdir=/home/[user]/.local/share/containers/storage/overlay/l/EEKIGRQDW3H3E6QG5QF4CUUYMP,upperdir=/home/[user]/.local/share/containers/storage/overlay/2f9acca52de42ebfaa791510efbf94204a4b7cb0c321b07b4a849e7715493b3e/diff,workdir=/home/[user]/.local/share/containers/storage/overlay/2f9acca52de42ebfaa791510efbf94204a4b7cb0c321b07b4a849e7715493b3e/work 
DEBU[0000] Made network namespace at /var/run/user/1000/netns/cni-3d5962ac-3c52-d013-f11c-e3fdfc9fbbaa for container c9f5a1379783034fc25c8ce4359794254fbcad52761ed8eaae7f6586e04e8961 
DEBU[0000] mounted container "c9f5a1379783034fc25c8ce4359794254fbcad52761ed8eaae7f6586e04e8961" at "/home/[user]/.local/share/containers/storage/overlay/2f9acca52de42ebfaa791510efbf94204a4b7cb0c321b07b4a849e7715493b3e/merged" 
DEBU[0000] slirp4netns command: /usr/bin/slirp4netns --disable-host-loopback --mtu 65520 --enable-sandbox --enable-seccomp -c -e 3 -r 4 --netns-type=path /var/run/user/1000/netns/cni-3d5962ac-3c52-d013-f11c-e3fdfc9fbbaa tap0 
DEBU[0000] Tearing down network namespace at /var/run/user/1000/netns/cni-3d5962ac-3c52-d013-f11c-e3fdfc9fbbaa for container c9f5a1379783034fc25c8ce4359794254fbcad52761ed8eaae7f6586e04e8961 
DEBU[0000] Cleaning up container c9f5a1379783034fc25c8ce4359794254fbcad52761ed8eaae7f6586e04e8961 
DEBU[0000] Network is already cleaned up, skipping...                                             
DEBU[0000] Container c9f5a1379783034fc25c8ce4359794254fbcad52761ed8eaae7f6586e04e8961 storage is already unmounted, skipping... 
DEBU[0000] Removing container c9f5a1379783034fc25c8ce4359794254fbcad52761ed8eaae7f6586e04e8961 
DEBU[0000] Removing all exec sessions for container c9f5a1379783034fc25c8ce4359794254fbcad52761ed8eaae7f6586e04e8961 
DEBU[0000] Cleaning up container c9f5a1379783034fc25c8ce4359794254fbcad52761ed8eaae7f6586e04e8961 
DEBU[0000] Network is already cleaned up, skipping...   
DEBU[0000] Container c9f5a1379783034fc25c8ce4359794254fbcad52761ed8eaae7f6586e04e8961 storage is already unmounted, skipping... 
DEBU[0000] Container c9f5a1379783034fc25c8ce4359794254fbcad52761ed8eaae7f6586e04e8961 storage is already unmounted, skipping... 
DEBU[0000] ExitCode msg: "error resolving storage path for container c9f5a1379783034fc25c8ce4359794254fbcad52761ed8eaae7f6586e04e8961: lstat /home/[user]/.local/share/containers/storage/overlay/2f9acca52de42ebfaa791510efbf94204a4b7cb0c321b07b4a849e7715493b3e/merged: invalid argument" 
Error: error resolving storage path for container c9f5a1379783034fc25c8ce4359794254fbcad52761ed8eaae7f6586e04e8961: lstat /home/[user]/.local/share/containers/storage/overlay/2f9acca52de42ebfaa791510efbf94204a4b7cb0c321b07b4a849e7715493b3e/merged: invalid argument
[...]

As you can see, the 'broken' one never reaches Created root filesystem for container, could something be silently failing?

@rhatdan
Copy link
Member

rhatdan commented Feb 2, 2021

I believe this is fixed in podman 3.0. With newer kernels, we should be able to get rootless overlay working without fuse-overlay as well.

@rhatdan rhatdan closed this as completed Feb 2, 2021
@Ristovski
Copy link
Author

@rhatdan I just confirmed this still happens with podman 3.2.1. Any ideas?

@Ristovski
Copy link
Author

Just recreated this on a fresh build of podman 3.3.0 as well.

$ zcat /proc/config.gz |grep -i overlay
CONFIG_OVERLAY_FS=m
CONFIG_OVERLAY_FS_REDIRECT_DIR=y
# CONFIG_OVERLAY_FS_REDIRECT_ALWAYS_FOLLOW is not set
CONFIG_OVERLAY_FS_INDEX=y
CONFIG_OVERLAY_FS_XINO_AUTO=y
CONFIG_OVERLAY_FS_METACOPY=y
Debug log 
INFO[0000] ./podman filtering at log level debug        
DEBU[0000] Called run.PersistentPreRunE(./podman --log-level=debug run --rm -it hello-world) 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf" 
DEBU[0000] Using conmon: "/usr/libexec/podman/conmon"   
DEBU[0000] Initializing boltdb state at /home/rafael/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/rafael/.local/share/containers/storage 
DEBU[0000] Using run root /var/run/user/1000/containers 
DEBU[0000] Using static dir /home/rafael/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /var/run/user/1000/libpod/tmp  
DEBU[0000] Using volume path /home/rafael/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] Not configuring container store              
DEBU[0000] Initializing event backend file              
DEBU[0000] configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Found CNI network podman (type=bridge) at /home/rafael/.config/cni/net.d/87-podman.conflist 
DEBU[0000] Default CNI network name podman is unchangeable 
WARN[0000] "/" is not a shared mount, this could cause issues or missing mounts with rootless containers 
INFO[0000] ./podman filtering at log level debug        
DEBU[0000] Called run.PersistentPreRunE(./podman --log-level=debug run --rm -it hello-world) 
DEBU[0000] cached value indicated that overlay is not supported 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf" 
DEBU[0000] cached value indicated that overlay is not supported 
DEBU[0000] Using conmon: "/usr/libexec/podman/conmon"   
DEBU[0000] Initializing boltdb state at /home/rafael/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/rafael/.local/share/containers/storage 
DEBU[0000] Using run root /var/run/user/1000/containers 
DEBU[0000] Using static dir /home/rafael/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /var/run/user/1000/libpod/tmp  
DEBU[0000] Using volume path /home/rafael/.local/share/containers/storage/volumes 
DEBU[0000] cached value indicated that overlay is not supported 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Initializing event backend file              
DEBU[0000] configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Found CNI network podman (type=bridge) at /home/rafael/.config/cni/net.d/87-podman.conflist 
DEBU[0000] Default CNI network name podman is unchangeable 
DEBU[0000] Podman detected system restart - performing state refresh 
INFO[0000] Setting parallel job count to 13             
DEBU[0000] Pulling image hello-world (policy: missing)  
DEBU[0000] Looking up image "hello-world" in local containers storage 
DEBU[0000] Trying "hello-world" ...                     
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" 
DEBU[0000] Trying "localhost/hello-world:latest" ...    
DEBU[0000] Trying "docker.io/library/hello-world:latest" ... 
DEBU[0000] Trying "docker.io/library/hello-world:latest" ... 
DEBU[0000] Attempting to pull candidate docker.io/library/hello-world:latest for hello-world 
DEBU[0000] parsed reference into "[overlay@/home/rafael/.local/share/containers/storage+/var/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]docker.io/library/hello-world:latest" 
DEBU[0000] Resolving "hello-world" using unqualified-search registries (/etc/containers/registries.conf) 
Resolving "hello-world" using unqualified-search registries (/etc/containers/registries.conf)
Trying to pull docker.io/library/hello-world:latest...
DEBU[0000] Copying source image //hello-world:latest to destination image [overlay@/home/rafael/.local/share/containers/storage+/var/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]docker.io/library/hello-world:latest 
DEBU[0000] Trying to access "mirror.gcr.io/library/hello-world:latest" 
DEBU[0000] No credentials for mirror.gcr.io found       
DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration 
DEBU[0000]  No signature storage configuration found for mirror.gcr.io/library/hello-world:latest, using built-in default file:///home/rafael/.local/share/containers/sigstore 
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/mirror.gcr.io 
DEBU[0000] GET https://mirror.gcr.io/v2/                
DEBU[0000] Ping https://mirror.gcr.io/v2/ status 401    
DEBU[0000] GET https://mirror.gcr.io/v2/token?scope=repository%3Alibrary%2Fhello-world%3Apull&service=mirror.gcr.io 
DEBU[0000] GET https://mirror.gcr.io/v2/library/hello-world/manifests/latest 
DEBU[0001] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.list.v2+json" 
DEBU[0001] Using blob info cache at /home/rafael/.local/share/containers/cache/blob-info-cache-v1.boltdb 
DEBU[0001] Source is a manifest list; copying (only) instance sha256:1b26826f602946860c279fce658f31050cff2c596583af237d971f4629b57792 for current system 
DEBU[0001] GET https://mirror.gcr.io/v2/library/hello-world/manifests/sha256:1b26826f602946860c279fce658f31050cff2c596583af237d971f4629b57792 
DEBU[0001] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.v2+json" 
DEBU[0001] IsRunningImageAllowed for image docker:docker.io/library/hello-world:latest 
DEBU[0001]  Using default policy section                
DEBU[0001]  Requirement 0: allowed                      
DEBU[0001] Overall: allowed                             
DEBU[0001] Downloading /v2/library/hello-world/blobs/sha256:d1165f2212346b2bab48cb01c1e39ee8ad1be46b87873d9ca7a4e434980a7726 
DEBU[0001] GET https://mirror.gcr.io/v2/library/hello-world/blobs/sha256:d1165f2212346b2bab48cb01c1e39ee8ad1be46b87873d9ca7a4e434980a7726 
Getting image source signatures
DEBU[0002] Reading /home/rafael/.local/share/containers/sigstore/library/hello-world@sha256=1b26826f602946860c279fce658f31050cff2c596583af237d971f4629b57792/signature-1 
DEBU[0002] Manifest has MIME type application/vnd.docker.distribution.manifest.v2+json, ordered candidate list [application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v1+json] 
DEBU[0002] ... will first try using the original manifest unmodified 
DEBU[0002] Failed to retrieve partial blob: blob type not supported for partial retrieval 
DEBU[0002] Downloading /v2/library/hello-world/blobs/sha256:b8dfde127a2919ff59ad3fd4a0776de178a555a76fff77a506e128aea3ed41e3 
DEBU[0002] GET https://mirror.gcr.io/v2/library/hello-world/blobs/sha256:b8dfde127a2919ff59ad3fd4a0776de178a555a76fff77a506e128aea3ed41e3 
DEBU[0002] Detected compression format gzip             
DEBU[0002] Using original blob without modification     
Copying blob b8dfde127a29 done  
Copying blob b8dfde127a29 done  
DEBU[0002] No compression detected                      
DEBU[0002] Using original blob without modification     
Copying config d1165f2212 done  
Writing manifest to image destination
Storing signatures
DEBU[0002] setting image creation date to 2021-03-05 23:25:25.230064203 +0000 UTC 
DEBU[0003] created new image ID "d1165f2212346b2bab48cb01c1e39ee8ad1be46b87873d9ca7a4e434980a7726" 
DEBU[0003] saved image metadata "{\"signatures-sizes\":{\"sha256:1b26826f602946860c279fce658f31050cff2c596583af237d971f4629b57792\":[]}}" 
DEBU[0003] set names of image "d1165f2212346b2bab48cb01c1e39ee8ad1be46b87873d9ca7a4e434980a7726" to [docker.io/library/hello-world:latest] 
DEBU[0003] Pulled candidate docker.io/library/hello-world:latest successfully 
DEBU[0003] Looking up image "d1165f2212346b2bab48cb01c1e39ee8ad1be46b87873d9ca7a4e434980a7726" in local containers storage 
DEBU[0003] Trying "d1165f2212346b2bab48cb01c1e39ee8ad1be46b87873d9ca7a4e434980a7726" ... 
DEBU[0003] parsed reference into "[overlay@/home/rafael/.local/share/containers/storage+/var/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@d1165f2212346b2bab48cb01c1e39ee8ad1be46b87873d9ca7a4e434980a7726" 
DEBU[0003] Found image "d1165f2212346b2bab48cb01c1e39ee8ad1be46b87873d9ca7a4e434980a7726" as "d1165f2212346b2bab48cb01c1e39ee8ad1be46b87873d9ca7a4e434980a7726" in local containers storage 
DEBU[0003] Found image "d1165f2212346b2bab48cb01c1e39ee8ad1be46b87873d9ca7a4e434980a7726" as "d1165f2212346b2bab48cb01c1e39ee8ad1be46b87873d9ca7a4e434980a7726" in local containers storage ([overlay@/home/rafael/.local/share/containers/storage+/var/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@d1165f2212346b2bab48cb01c1e39ee8ad1be46b87873d9ca7a4e434980a7726) 
DEBU[0003] Looking up image "hello-world" in local containers storage 
DEBU[0003] Trying "hello-world" ...                     
DEBU[0003] Trying "localhost/hello-world:latest" ...    
DEBU[0003] Trying "docker.io/library/hello-world:latest" ... 
DEBU[0003] parsed reference into "[overlay@/home/rafael/.local/share/containers/storage+/var/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@d1165f2212346b2bab48cb01c1e39ee8ad1be46b87873d9ca7a4e434980a7726" 
DEBU[0003] Found image "hello-world" as "docker.io/library/hello-world:latest" in local containers storage 
DEBU[0003] Found image "hello-world" as "docker.io/library/hello-world:latest" in local containers storage ([overlay@/home/rafael/.local/share/containers/storage+/var/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@d1165f2212346b2bab48cb01c1e39ee8ad1be46b87873d9ca7a4e434980a7726) 
DEBU[0003] Inspecting image d1165f2212346b2bab48cb01c1e39ee8ad1be46b87873d9ca7a4e434980a7726 
DEBU[0003] exporting opaque data as blob "sha256:d1165f2212346b2bab48cb01c1e39ee8ad1be46b87873d9ca7a4e434980a7726" 
DEBU[0003] exporting opaque data as blob "sha256:d1165f2212346b2bab48cb01c1e39ee8ad1be46b87873d9ca7a4e434980a7726" 
DEBU[0003] exporting opaque data as blob "sha256:d1165f2212346b2bab48cb01c1e39ee8ad1be46b87873d9ca7a4e434980a7726" 
DEBU[0003] exporting opaque data as blob "sha256:d1165f2212346b2bab48cb01c1e39ee8ad1be46b87873d9ca7a4e434980a7726" 
DEBU[0003] Looking up image "hello-world" in local containers storage 
DEBU[0003] Trying "hello-world" ...                     
DEBU[0003] Trying "localhost/hello-world:latest" ...    
DEBU[0003] Trying "docker.io/library/hello-world:latest" ... 
DEBU[0003] parsed reference into "[overlay@/home/rafael/.local/share/containers/storage+/var/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@d1165f2212346b2bab48cb01c1e39ee8ad1be46b87873d9ca7a4e434980a7726" 
DEBU[0003] Found image "hello-world" as "docker.io/library/hello-world:latest" in local containers storage 
DEBU[0003] Found image "hello-world" as "docker.io/library/hello-world:latest" in local containers storage ([overlay@/home/rafael/.local/share/containers/storage+/var/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@d1165f2212346b2bab48cb01c1e39ee8ad1be46b87873d9ca7a4e434980a7726) 
DEBU[0003] Inspecting image d1165f2212346b2bab48cb01c1e39ee8ad1be46b87873d9ca7a4e434980a7726 
DEBU[0003] exporting opaque data as blob "sha256:d1165f2212346b2bab48cb01c1e39ee8ad1be46b87873d9ca7a4e434980a7726" 
DEBU[0003] exporting opaque data as blob "sha256:d1165f2212346b2bab48cb01c1e39ee8ad1be46b87873d9ca7a4e434980a7726" 
DEBU[0003] exporting opaque data as blob "sha256:d1165f2212346b2bab48cb01c1e39ee8ad1be46b87873d9ca7a4e434980a7726" 
DEBU[0003] exporting opaque data as blob "sha256:d1165f2212346b2bab48cb01c1e39ee8ad1be46b87873d9ca7a4e434980a7726" 
DEBU[0003] Inspecting image d1165f2212346b2bab48cb01c1e39ee8ad1be46b87873d9ca7a4e434980a7726 
DEBU[0003] using systemd mode: false                    
DEBU[0003] No hostname set; container's hostname will default to runtime default 
DEBU[0003] Loading seccomp profile from "/usr/share/containers/seccomp.json" 
DEBU[0003] Allocated lock 0 for container 373a7fdbbd2f1757c24bc124943f983e1fef73406677b1167348d46788c88da4 
DEBU[0003] parsed reference into "[overlay@/home/rafael/.local/share/containers/storage+/var/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@d1165f2212346b2bab48cb01c1e39ee8ad1be46b87873d9ca7a4e434980a7726" 
DEBU[0003] exporting opaque data as blob "sha256:d1165f2212346b2bab48cb01c1e39ee8ad1be46b87873d9ca7a4e434980a7726" 
DEBU[0003] created container "373a7fdbbd2f1757c24bc124943f983e1fef73406677b1167348d46788c88da4" 
DEBU[0003] container "373a7fdbbd2f1757c24bc124943f983e1fef73406677b1167348d46788c88da4" has work directory "/home/rafael/.local/share/containers/storage/overlay-containers/373a7fdbbd2f1757c24bc124943f983e1fef73406677b1167348d46788c88da4/userdata" 
DEBU[0003] container "373a7fdbbd2f1757c24bc124943f983e1fef73406677b1167348d46788c88da4" has run directory "/var/run/user/1000/containers/overlay-containers/373a7fdbbd2f1757c24bc124943f983e1fef73406677b1167348d46788c88da4/userdata" 
DEBU[0003] Handling terminal attach                     
DEBU[0003] [graphdriver] trying provided driver "overlay" 
DEBU[0003] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0003] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0003] overlay: mount_data=,lowerdir=/home/rafael/.local/share/containers/storage/overlay/l/4XMXZ6S2FO7SATWXTH6ZNBJXGW,upperdir=/home/rafael/.local/share/containers/storage/overlay/0c993b862c7ab42d658fe7205e6b9013e044787ca7ac663405d15bdd5e7f2059/diff,workdir=/home/rafael/.local/share/containers/storage/overlay/0c993b862c7ab42d658fe7205e6b9013e044787ca7ac663405d15bdd5e7f2059/work,volatile 
DEBU[0003] Made network namespace at /var/run/user/1000/netns/cni-3b51948c-b587-4736-b55f-2a5fafe46c9c for container 373a7fdbbd2f1757c24bc124943f983e1fef73406677b1167348d46788c88da4 
DEBU[0003] mounted container "373a7fdbbd2f1757c24bc124943f983e1fef73406677b1167348d46788c88da4" at "/home/rafael/.local/share/containers/storage/overlay/0c993b862c7ab42d658fe7205e6b9013e044787ca7ac663405d15bdd5e7f2059/merged" 
DEBU[0003] slirp4netns command: /usr/bin/slirp4netns --disable-host-loopback --mtu=65520 --enable-sandbox --enable-seccomp -c -e 3 -r 4 --netns-type=path /var/run/user/1000/netns/cni-3b51948c-b587-4736-b55f-2a5fafe46c9c tap0 
DEBU[0003] Tearing down network namespace at /var/run/user/1000/netns/cni-3b51948c-b587-4736-b55f-2a5fafe46c9c for container 373a7fdbbd2f1757c24bc124943f983e1fef73406677b1167348d46788c88da4 
DEBU[0003] Cleaning up container 373a7fdbbd2f1757c24bc124943f983e1fef73406677b1167348d46788c88da4 
DEBU[0003] Network is already cleaned up, skipping...                                             
DEBU[0003] Container 373a7fdbbd2f1757c24bc124943f983e1fef73406677b1167348d46788c88da4 storage is already unmounted, skipping... 
DEBU[0003] Removing container 373a7fdbbd2f1757c24bc124943f983e1fef73406677b1167348d46788c88da4                                  
DEBU[0003] Removing all exec sessions for container 373a7fdbbd2f1757c24bc124943f983e1fef73406677b1167348d46788c88da4 
DEBU[0003] Cleaning up container 373a7fdbbd2f1757c24bc124943f983e1fef73406677b1167348d46788c88da4 
DEBU[0003] Network is already cleaned up, skipping...   
DEBU[0003] Container 373a7fdbbd2f1757c24bc124943f983e1fef73406677b1167348d46788c88da4 storage is already unmounted, skipping... 
DEBU[0003] Container 373a7fdbbd2f1757c24bc124943f983e1fef73406677b1167348d46788c88da4 storage is already unmounted, skipping... 
DEBU[0004] ExitCode msg: "error resolving storage path for container 373a7fdbbd2f1757c24bc124943f983e1fef73406677b1167348d46788c88da4: lstat /home/rafael/.local/share/containers/storage/overlay/0c993b862c7ab42d658fe7205e6b9013e044787ca7ac663405d15bdd5e7f2059/merged: invalid argument" 
Error: error resolving storage path for container 373a7fdbbd2f1757c24bc124943f983e1fef73406677b1167348d46788c88da4: lstat /home/rafael/.local/share/containers/storage/overlay/0c993b862c7ab42d658fe7205e6b9013e044787ca7ac663405d15bdd5e7f2059/merged: invalid argument

@rhatdan
Copy link
Member

rhatdan commented Aug 3, 2021

@giuseppe PTAL

@Ristovski
Copy link
Author

Tested with kernel 5.14-rc4, can confirm that it still doesn't work with rootless mode.

@Ristovski
Copy link
Author

Isolated this to a fuse-overlayfs issue: containers/fuse-overlayfs#311 (comment)

@giuseppe
Copy link
Member

giuseppe commented Aug 4, 2021

let's follow up on the fuse-overlayfs issue

@giuseppe
Copy link
Member

giuseppe commented Aug 4, 2021

nevermind my last comment. The error:

Error: error resolving storage path for container 373a7fdbbd2f1757c24bc124943f983e1fef73406677b1167348d46788c88da4: lstat /home/rafael/.local/share/containers/storage/overlay/0c993b862c7ab42d658fe7205e6b9013e044787ca7ac663405d15bdd5e7f2059/merged: invalid argument

happens in Podman.

Could you run something like:

podman unshare strace -s 1000 -e lstat -Z -f podman container run --rm -it hello-world

and show the output?

What file system are you using for /home/rafael?

Since both this issue and the one reported for fuse-overlayfs happen on Gentoo, I guess it is something related to it

@Ristovski
Copy link
Author

Ristovski commented Aug 4, 2021 •
edited
Loading

@giuseppe Here is the strace log, doesn't seem to have anything interesting. My whole drive is using ext4.

I have really no clue what could be causing this. I've tested multiple fuse-overlayfs versions, including master.
This is the one I am running at the moment:

$ fuse-overlayfs --version
fuse-overlayfs: version 1.5
FUSE library version 3.10.4
using FUSE kernel interface version 7.31
fusermount3 version: 3.10.4

I have also attached a strace of manually running a fuse-overlayfs command. The same error happens when running it with sudo as well.

Strace log 
strace: Process 12157 attached
strace: Process 12158 attached
strace: Process 12159 attached
strace: Process 12160 attached
strace: Process 12161 attached
strace: Process 12162 attached
[pid 12160] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=12156, si_uid=0} ---
[pid 12156] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=12156, si_uid=0} ---
strace: Process 12163 attached
[pid 12156] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=12156, si_uid=0} ---
[pid 12156] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=12156, si_uid=0} ---
[pid 12156] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=12156, si_uid=0} ---
[pid 12156] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=12156, si_uid=0} ---
[pid 12156] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=12156, si_uid=0} ---
[pid 12159] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=12156, si_uid=0} ---
[pid 12162] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=12156, si_uid=0} ---
[pid 12156] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=12156, si_uid=0} ---
[pid 12156] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=12156, si_uid=0} ---
[pid 12156] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=12156, si_uid=0} ---
strace: Process 12164 attached
[pid 12164] +++ exited with 0 +++
[pid 12163] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=12164, si_uid=0, si_status=0, si_utime=0, si_stime=0} ---
strace: Process 12165 attached
strace: Process 12166 attached
strace: Process 12167 attached
strace: Process 12168 attached
strace: Process 12169 attached
strace: Process 12170 attached
[pid 12169] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=12165, si_uid=0} ---
strace: Process 12171 attached
[pid 12170] +++ exited with 0 +++
[pid 12168] +++ exited with 0 +++
[pid 12167] +++ exited with 0 +++
[pid 12171] +++ exited with 0 +++
[pid 12169] +++ exited with 0 +++
[pid 12165] +++ exited with 0 +++
[pid 12163] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=12165, si_uid=0, si_status=0, si_utime=0, si_stime=0} ---
strace: Process 12172 attached
strace: Process 12173 attached
strace: Process 12174 attached
strace: Process 12175 attached
strace: Process 12176 attached
strace: Process 12177 attached
[pid 12177] +++ exited with 0 +++
[pid 12176] +++ exited with 0 +++
[pid 12175] +++ exited with 0 +++
[pid 12173] +++ exited with 0 +++
[pid 12174] +++ exited with 0 +++
[pid 12172] +++ exited with 0 +++
[pid 12163] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=12172, si_uid=0, si_status=0, si_utime=0, si_stime=0} ---
strace: Process 12178 attached
strace: Process 12179 attached
strace: Process 12180 attached
strace: Process 12181 attached
strace: Process 12182 attached
strace: Process 12183 attached
[pid 12183] +++ exited with 0 +++
[pid 12182] +++ exited with 0 +++
[pid 12179] +++ exited with 0 +++
[pid 12181] +++ exited with 0 +++
[pid 12180] +++ exited with 0 +++
[pid 12178] +++ exited with 0 +++
[pid 12163] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=12178, si_uid=0, si_status=0, si_utime=0, si_stime=0} ---
strace: Process 12184 attached
strace: Process 12185 attached
strace: Process 12186 attached
strace: Process 12187 attached
strace: Process 12188 attached
strace: Process 12189 attached
[pid 12189] +++ exited with 0 +++
[pid 12188] +++ exited with 0 +++
[pid 12186] +++ exited with 0 +++
[pid 12185] +++ exited with 0 +++
[pid 12187] +++ exited with 0 +++
[pid 12184] +++ exited with 0 +++
[pid 12159] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=12184, si_uid=0, si_status=0, si_utime=0, si_stime=0} ---
strace: Process 12190 attached
[pid 12159] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=12156, si_uid=0} ---
[pid 12162] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=12156, si_uid=0} ---
[pid 12158] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=12156, si_uid=0} ---
[pid 12162] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=12156, si_uid=0} ---
[pid 12162] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=12156, si_uid=0} ---
[pid 12162] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=12156, si_uid=0} ---
[pid 12158] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=12156, si_uid=0} ---
[pid 12158] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=12156, si_uid=0} ---
[pid 12190] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=12156, si_uid=0} ---
[pid 12159] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=12156, si_uid=0} ---
[pid 12158] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=12156, si_uid=0} ---
[pid 12162] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=12156, si_uid=0} ---
[pid 12158] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=12156, si_uid=0} ---
[pid 12159] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=12156, si_uid=0} ---
[pid 12159] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=12156, si_uid=0} ---
[pid 12159] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=12156, si_uid=0} ---
[pid 12159] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=12156, si_uid=0} ---
[pid 12159] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=12156, si_uid=0} ---
strace: Process 12191 attached
[pid 12158] +++ exited with 0 +++
strace: Process 12192 attached
[pid 12163] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=12156, si_uid=0} ---
strace: Process 12193 attached
[pid 12190] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=12156, si_uid=0} ---
strace: Process 12194 attached
[pid 12190] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=12156, si_uid=0} ---
[pid 12194] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=12156, si_uid=0} ---
[pid 12190] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=12156, si_uid=0} ---
[pid 12191] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=12156, si_uid=0} ---
[pid 12193] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=12156, si_uid=0} ---
[pid 12190] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=12156, si_uid=0} ---
[pid 12191] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=12156, si_uid=0} ---
strace: Process 12195 attached
strace: Process 12197 attached
[pid 12192] +++ exited with 0 +++
[pid 12194] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=12192, si_uid=0, si_status=0, si_utime=0, si_stime=0} ---
[pid 12195] +++ exited with 0 +++
[pid 12191] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=12195, si_uid=0, si_status=0, si_utime=0, si_stime=0} ---
strace: Process 12198 attached
strace: Process 12199 attached
[pid 12199] +++ exited with 0 +++
[pid 12198] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=12199, si_uid=0, si_status=0, si_utime=0, si_stime=0} ---
[pid 12159] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=12156, si_uid=0} ---
[pid 12159] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=12156, si_uid=0} ---
[pid 12159] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=12156, si_uid=0} ---
[pid 12159] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=12156, si_uid=0} ---
strace: Process 12200 attached
[pid 12200] +++ exited with 0 +++
[pid 12194] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=12200, si_uid=0, si_status=0, si_utime=0, si_stime=0} ---
[pid 12197] +++ exited with 0 +++
[pid 12194] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=12156, si_uid=0} ---
[pid 12159] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=12156, si_uid=0} ---
[pid 12159] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=12156, si_uid=0} ---
Error: error resolving storage path for container 56d6dbe005b4bf1433f00d48a9eac056c4a72f3789cd624d022f1fb4e945dc22: lstat /home/rafael/.local/share/containers/storage/overlay/be5bd1b24dd21a2b885ab16b789a179fce4bfa523386191aa572426312e21b5e/merged: invalid argument
[pid 12191] +++ exited with 126 +++
[pid 12166] +++ exited with 126 +++
[pid 12162] +++ exited with 126 +++
[pid 12160] +++ exited with 126 +++
[pid 12161] +++ exited with 126 +++
[pid 12157] +++ exited with 126 +++
[pid 12194] +++ exited with 126 +++
[pid 12163] +++ exited with 126 +++
[pid 12193] +++ exited with 126 +++
[pid 12190] +++ exited with 126 +++
[pid 12159] +++ exited with 126 +++
[pid 12156] +++ exited with 126 +++
+++ exited with 0 +++
Error: exit status 126
Manual fuse-overlayfs invocation 
execve("/usr/bin/fuse-overlayfs", ["fuse-overlayfs", "-o", "lowerdir=lowerdir,upperdir=upper"..., "merged", "-d"], 0x7fff4ac253d0 /* 135 vars */) = 0
access("/etc/suid-debug", F_OK)         = -1 ENOENT (No such file or directory)
brk(NULL)                               = 0x561cbf482000
fcntl(0, F_GETFD)                       = 0
fcntl(1, F_GETFD)                       = 0
fcntl(2, F_GETFD)                       = 0
access("/etc/suid-debug", F_OK)         = -1 ENOENT (No such file or directory)
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=361870, ...}) = 0
mmap(NULL, 361870, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f6a0204c000
close(3)                                = 0
openat(AT_FDCWD, "/usr/lib64/libfuse3.so.3", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\340}\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=298616, ...}) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6a0204a000
mmap(NULL, 300960, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f6a02000000
mprotect(0x7f6a02007000, 196608, PROT_NONE) = 0
mmap(0x7f6a02007000, 151552, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x7000) = 0x7f6a02007000
mmap(0x7f6a0202c000, 40960, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2c000) = 0x7f6a0202c000
mmap(0x7f6a02037000, 77824, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x36000) = 0x7f6a02037000
close(3)                                = 0
openat(AT_FDCWD, "/lib64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0pm\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=155096, ...}) = 0
mmap(NULL, 140448, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f6a01fdd000
mmap(0x7f6a01fe3000, 73728, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6000) = 0x7f6a01fe3000
mmap(0x7f6a01ff5000, 20480, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x18000) = 0x7f6a01ff5000
mmap(0x7f6a01ffa000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1c000) = 0x7f6a01ffa000
mmap(0x7f6a01ffc000, 13472, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f6a01ffc000
close(3)                                = 0
openat(AT_FDCWD, "/lib64/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0000\21\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=18416, ...}) = 0
mmap(NULL, 20752, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f6a01fd7000
mmap(0x7f6a01fd8000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1000) = 0x7f6a01fd8000
mmap(0x7f6a01fda000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f6a01fda000
mmap(0x7f6a01fdb000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f6a01fdb000
close(3)                                = 0
openat(AT_FDCWD, "/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320?\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1877768, ...}) = 0
mmap(NULL, 1890152, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f6a01e09000
mprotect(0x7f6a01e2b000, 1712128, PROT_NONE) = 0
mmap(0x7f6a01e2b000, 1409024, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x22000) = 0x7f6a01e2b000
mmap(0x7f6a01f83000, 299008, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17a000) = 0x7f6a01f83000
mmap(0x7f6a01fcd000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1c3000) = 0x7f6a01fcd000
mmap(0x7f6a01fd3000, 14184, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f6a01fd3000
close(3)                                = 0
openat(AT_FDCWD, "/lib64/librt.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240#\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=35528, ...}) = 0
mmap(NULL, 39904, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f6a01dff000
mmap(0x7f6a01e01000, 16384, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f6a01e01000
mmap(0x7f6a01e05000, 8192, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6000) = 0x7f6a01e05000
mmap(0x7f6a01e07000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x7000) = 0x7f6a01e07000
close(3)                                = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6a01dfd000
arch_prctl(ARCH_SET_FS, 0x7f6a01dfe380) = 0
mprotect(0x7f6a01fcd000, 16384, PROT_READ) = 0
mprotect(0x7f6a01ffa000, 4096, PROT_READ) = 0
mprotect(0x7f6a01e07000, 4096, PROT_READ) = 0
mprotect(0x7f6a01fdb000, 4096, PROT_READ) = 0
mprotect(0x7f6a02037000, 73728, PROT_READ) = 0
mprotect(0x561cbddd6000, 4096, PROT_READ) = 0
mprotect(0x7f6a020cd000, 4096, PROT_READ) = 0
munmap(0x7f6a0204c000, 361870)          = 0
set_tid_address(0x7f6a01dfe650)         = 13304
set_robust_list(0x7f6a01dfe660, 24)     = 0
rt_sigaction(SIGRTMIN, {sa_handler=0x7f6a01fe36d0, sa_mask=[], sa_flags=SA_RESTORER|SA_SIGINFO, sa_restorer=0x7f6a01ff1830}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {sa_handler=0x7f6a01fe37a0, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f6a01ff1830}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
brk(NULL)                               = 0x561cbf482000
brk(0x561cbf4a3000)                     = 0x561cbf4a3000
geteuid()                               = 1000
openat(AT_FDCWD, "/proc/sys/kernel/overflowuid", O_RDONLY) = 3
read(3, "65534\n", 255)                 = 6
close(3)                                = 0
openat(AT_FDCWD, "/proc/sys/kernel/overflowgid", O_RDONLY) = 3
read(3, "65534\n", 255)                 = 6
close(3)                                = 0
geteuid()                               = 1000
getegid()                               = 1000
getcwd("/tmp/overlaytest", 4096)        = 17
lstat("/tmp/overlaytest/upperdir", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
prlimit64(0, RLIMIT_NOFILE, NULL, {rlim_cur=4*1024, rlim_max=500000}) = 0
prlimit64(0, RLIMIT_NOFILE, {rlim_cur=500000, rlim_max=500000}, NULL) = 0
mknodat(0, "1", S_IFCHR|0700, makedev(0, 0)) = -1 ENOTDIR (Not a directory)
write(2, "uid=unchanged\n", 14uid=unchanged
)         = 14
write(2, "gid=unchanged\n", 14gid=unchanged
)         = 14
write(2, "upperdir=/tmp/overlaytest/upperd"..., 35upperdir=/tmp/overlaytest/upperdir
) = 35
write(2, "workdir=workdir\n", 16workdir=workdir
)       = 16
write(2, "lowerdir=lowerdir\n", 18lowerdir=lowerdir
)     = 18
write(2, "mountpoint=merged\n", 18mountpoint=merged
)     = 18
write(2, "plugins=<none>\n", 15plugins=<none>
)        = 15
write(2, "fsync=enabled\n", 14fsync=enabled
)         = 14
openat(AT_FDCWD, "/usr/libexec/fuse-overlayfs", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = -1 ENOENT (No such file or directory)
getcwd("/tmp/overlaytest", 4096)        = 17
lstat("/tmp/overlaytest/lowerdir", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
openat(AT_FDCWD, "lowerdir", O_RDONLY|O_DIRECTORY) = 3
fgetxattr(3, "security.fuseoverlayfs.override_"..., 0x7fff327d8c90, 64) = -1 ENODATA (No data available)
fgetxattr(3, "user.fuseoverlayfs.override_stat", 0x7fff327d8c90, 64) = -1 ENODATA (No data available)
fgetxattr(3, "user.containers.override_stat", 0x7fff327d8c90, 64) = -1 ENODATA (No data available)
lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=335872, ...}) = 0
lstat("/tmp/overlaytest", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/tmp/overlaytest/upperdir", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
openat(AT_FDCWD, "/tmp/overlaytest/upperdir", O_RDONLY|O_DIRECTORY) = 4
fgetxattr(4, "security.fuseoverlayfs.override_"..., 0x7fff327d8c90, 64) = -1 ENODATA (No data available)
fgetxattr(4, "user.fuseoverlayfs.override_stat", 0x7fff327d8c90, 64) = -1 ENODATA (No data available)
fgetxattr(4, "user.containers.override_stat", 0x7fff327d8c90, 64) = -1 ENODATA (No data available)
newfstatat(4, "/.wh.", 0x7fff327d46f0, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)
openat2(4, ".", {flags=O_RDONLY|O_NONBLOCK|O_NOFOLLOW, resolve=RESOLVE_IN_ROOT}, 24) = 5
fgetxattr(5, "trusted.overlay.origin", 0x561cbf48f2e0, 4096) = -1 ENODATA (No data available)
fgetxattr(5, "user.fuseoverlayfs.origin", 0x561cbf48f2e0, 4096) = -1 ENODATA (No data available)
close(5)                                = 0
newfstatat(3, "/.wh.", 0x7fff327d46f0, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)
openat2(3, ".", {flags=O_RDONLY|O_NONBLOCK|O_NOFOLLOW, resolve=RESOLVE_IN_ROOT}, 24) = 5
fgetxattr(5, "trusted.overlay.origin", 0x561cbf48f2e0, 4096) = -1 ENODATA (No data available)
fgetxattr(5, "user.fuseoverlayfs.origin", 0x561cbf48f2e0, 4096) = -1 ENODATA (No data available)
close(5)                                = 0
newfstatat(4, ".wh.", 0x7fff327d5a60, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)
openat2(4, ".", {flags=O_RDONLY|O_DIRECTORY, resolve=RESOLVE_IN_ROOT}, 24) = 5
fstat(5, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
fcntl(5, F_GETFL)                       = 0x18000 (flags O_RDONLY|O_LARGEFILE|O_DIRECTORY)
fcntl(5, F_SETFD, FD_CLOEXEC)           = 0
getdents64(5, 0x561cbf48f340 /* 2 entries */, 32768) = 48
getdents64(5, 0x561cbf48f340 /* 0 entries */, 32768) = 0
openat2(4, ".", {flags=O_RDONLY|O_NONBLOCK|O_NOFOLLOW, resolve=RESOLVE_IN_ROOT}, 24) = 7
fgetxattr(7, "trusted.overlay.opaque", 0x7fff327d5bb0, 16) = -1 ENODATA (No data available)
close(7)                                = 0
openat2(4, ".", {flags=O_RDONLY|O_NONBLOCK|O_NOFOLLOW, resolve=RESOLVE_IN_ROOT}, 24) = 7
fgetxattr(7, "user.fuseoverlayfs.opaque", 0x7fff327d5bb0, 16) = -1 ENODATA (No data available)
close(7)                                = 0
newfstatat(4, "./.wh..wh..opq", 0x7fff327d49e0, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)
close(5)                                = 0
newfstatat(3, ".wh.", 0x7fff327d5a60, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)
openat2(3, ".", {flags=O_RDONLY|O_DIRECTORY, resolve=RESOLVE_IN_ROOT}, 24) = 5
fstat(5, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
fcntl(5, F_GETFL)                       = 0x18000 (flags O_RDONLY|O_LARGEFILE|O_DIRECTORY)
fcntl(5, F_SETFD, FD_CLOEXEC)           = 0
getdents64(5, 0x561cbf48f340 /* 2 entries */, 32768) = 48
getdents64(5, 0x561cbf48f340 /* 0 entries */, 32768) = 0
openat2(3, ".", {flags=O_RDONLY|O_NONBLOCK|O_NOFOLLOW, resolve=RESOLVE_IN_ROOT}, 24) = 7
fgetxattr(7, "trusted.overlay.opaque", 0x7fff327d5bb0, 16) = -1 ENODATA (No data available)
close(7)                                = 0
openat2(3, ".", {flags=O_RDONLY|O_NONBLOCK|O_NOFOLLOW, resolve=RESOLVE_IN_ROOT}, 24) = 7
fgetxattr(7, "user.fuseoverlayfs.opaque", 0x7fff327d5bb0, 16) = -1 ENODATA (No data available)
close(7)                                = 0
newfstatat(3, "./.wh..wh..opq", 0x7fff327d49e0, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)
close(5)                                = 0
getcwd("/tmp/overlaytest", 4096)        = 17
lstat("/tmp/overlaytest/workdir", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
mkdir("/tmp/overlaytest/workdir", 0700) = -1 EEXIST (File exists)
mkdir("/tmp/overlaytest/workdir/work", 0700) = -1 EEXIST (File exists)
openat(AT_FDCWD, "/tmp/overlaytest/workdir/work", O_RDONLY|O_DIRECTORY) = 5
dup(5)                                  = 7
fstat(7, {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0
fcntl(7, F_GETFL)                       = 0x18000 (flags O_RDONLY|O_LARGEFILE|O_DIRECTORY)
fcntl(7, F_SETFD, FD_CLOEXEC)           = 0
getdents64(7, 0x561cbf48f370 /* 2 entries */, 32768) = 48
getdents64(7, 0x561cbf48f370 /* 0 entries */, 32768) = 0
close(7)                                = 0
umask(000)                              = 022
write(2, "FUSE library version: 3.10.4\n", 29FUSE library version: 3.10.4
) = 29
getuid()                                = 1000
rt_sigaction(SIGHUP, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigaction(SIGHUP, {sa_handler=0x7f6a02023c20, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f6a01ff1830}, NULL, 8) = 0
rt_sigaction(SIGINT, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigaction(SIGINT, {sa_handler=0x7f6a02023c20, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f6a01ff1830}, NULL, 8) = 0
rt_sigaction(SIGTERM, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigaction(SIGTERM, {sa_handler=0x7f6a02023c20, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f6a01ff1830}, NULL, 8) = 0
rt_sigaction(SIGPIPE, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigaction(SIGPIPE, {sa_handler=0x7f6a02023c00, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f6a01ff1830}, NULL, 8) = 0
rt_sigaction(SIGUSR1, {sa_handler=0x561cbddc1b00, sa_mask=[USR1], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f6a01e41eb0}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
openat(AT_FDCWD, "/dev/null", O_RDWR)   = 7
close(7)                                = 0
stat("merged", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
openat(AT_FDCWD, "/dev/fuse", O_RDWR|O_CLOEXEC) = 7
getgid()                                = 1000
getuid()                                = 1000
mount("fuse-overlayfs", "merged", "fuse.fuse-overlayfs", MS_NOSUID|MS_NODEV, "default_permissions,fd=7,rootmod"...) = -1 EPERM (Operation not permitted)
close(7)                                = 0
socketpair(AF_UNIX, SOCK_STREAM, 0, [7, 8]) = 0
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f6a01dfe650) = 13305
close(7)                                = 0
recvmsg(8, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\0", iov_len=1}], msg_iovlen=1, msg_control=[{cmsg_len=20, cmsg_level=SOL_SOCKET, cmsg_type=SCM_RIGHTS, cmsg_data=[7]}], msg_controllen=24, msg_flags=0}, 0) = 1
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=13305, si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
close(8)                                = 0
wait4(13305, NULL, 0, NULL)             = 13305
fcntl(7, F_SETFD, FD_CLOEXEC)           = 0
chdir("/")                              = 0
mmap(NULL, 1056768, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6a01cfb000
read(7, "8\0\0\0\32\0\0\0\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1052672) = 56
write(2, "unique: 2, opcode: INIT (26), no"..., 60unique: 2, opcode: INIT (26), nodeid: 0, insize: 56, pid: 0
) = 60
write(2, "INIT: 7.34\n", 11INIT: 7.34
)            = 11
write(2, "flags=0x33fffffb\n", 17flags=0x33fffffb
)      = 17
write(2, "max_readahead=0x00020000\n", 25max_readahead=0x00020000
) = 25
write(2, "   INIT: 7.31\n", 14   INIT: 7.31
)         = 14
write(2, "   flags=0x0041f069\n", 20   flags=0x0041f069
)   = 20
write(2, "   max_readahead=0x00020000\n", 28   max_readahead=0x00020000
) = 28
write(2, "   max_write=0x00100000\n", 24   max_write=0x00100000
) = 24
write(2, "   max_background=0\n", 20   max_background=0
)   = 20
write(2, "   congestion_threshold=0\n", 26   congestion_threshold=0
) = 26
write(2, "   time_gran=1\n", 15   time_gran=1
)        = 15
write(2, "   unique: 2, success, outsize: "..., 35   unique: 2, success, outsize: 80
) = 35
writev(7, [{iov_base="P\0\0\0\0\0\0\0\2\0\0\0\0\0\0\0", iov_len=16}, {iov_base="\7\0\0\0\37\0\0\0\0\0\2\0i\360A\0\0\0\0\0\0\0\20\0\1\0\0\0\0\1\0\0"..., iov_len=64}], 2) = 80
pipe2([8, 9], O_NONBLOCK|O_CLOEXEC)     = 0
fcntl(8, F_SETPIPE_SZ, 1052672)         = -1 EPERM (Operation not permitted)
openat(AT_FDCWD, "/proc/sys/fs/pipe-max-size", O_RDONLY) = 10
read(10, "1048576\n", 31)               = 8
close(10)                               = 0
fcntl(8, F_SETPIPE_SZ, 1048576)         = 1048576
read(7, "8\0\0\0\3\0\0\0\4\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\350\3\0\0\350\3\0\0"..., 1052672) = 56
write(2, "unique: 4, opcode: GETATTR (3), "..., 66unique: 4, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 12220
) = 66
write(2, "ovl_getattr(ino=1)\n", 19ovl_getattr(ino=1)
)    = 19
write(2, "   unique: 4, error: -22 (Invali"..., 57   unique: 4, error: -22 (Invalid argument), outsize: 16
) = 57
writev(7, [{iov_base="\20\0\0\0\352\377\377\377\4\0\0\0\0\0\0\0", iov_len=16}], 1) = 16
read(7, 0x7f6a01cfb010, 1052672)        = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
strace: Process 13304 detached

@apollo13
Copy link

This issues was resolved for me by upgrading fuse-overlayfs to 1.7.1 -- you might wanna check with that one @Ristovski

@Ristovski
Copy link
Author

@apollo13 Indeed! This got fixed in containers/fuse-overlayfs#316, I forgot to bump this issue.

@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 21, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 21, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@giuseppe @apollo13 @Ristovski @rhatdan @afbjorklund @openshift-ci-robot