Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error: container create failed (no logs from conmon) #21032

Closed
grzegorzk opened this issue Dec 15, 2023 · 3 comments
Closed

Error: container create failed (no logs from conmon) #21032

grzegorzk opened this issue Dec 15, 2023 · 3 comments
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.

Comments

@grzegorzk
Copy link

grzegorzk commented Dec 15, 2023 •
edited
Loading

Issue Description

I can't run podman containers, all containers I tried fail with error

Steps to reproduce the issue

Steps to reproduce the issue

  1. podman pull python
  2. podman run -it python

Describe the results you received

Error: container create failed (no logs from conmon): conmon bytes "": readObjectStart: expect { or n, but found , error found in #0 byte of ...||..., bigger context ...||...

Describe the results you expected

Python prompt

podman info output

host:
  arch: amd64
  buildahVersion: 1.33.2
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: /usr/bin/conmon is owned by conmon 1:2.1.9-1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.9, commit: 3a9715d28cb4cf0e671dfbc4211d4458534db189'
  cpuUtilization:
    idlePercent: 96.94
    systemPercent: 0.66
    userPercent: 2.4
  cpus: 12
  databaseBackend: boltdb
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  freeLocks: 2022
  hostname: lenovo
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
  kernel: 6.6.7-arch1-1
  linkmode: dynamic
  logDriver: journald
  memFree: 115405082624
  memTotal: 134674116608
  networkBackend: cni
  networkBackendInfo:
    backend: cni
    dns: {}
    package: /usr/lib/cni/ is owned by cni-plugins 1.4.0-1
    path: /usr/lib/cni
  ociRuntime:
    name: crun
    package: /usr/bin/crun is owned by crun 1.12-1
    path: /usr/bin/crun
    version: |-
      crun version 1.12
      commit: ce429cb2e277d001c2179df1ac66a470f00802ae
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    exists: false
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: /usr/bin/slirp4netns is owned by slirp4netns 1.2.2-1
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 0
  swapTotal: 0
  uptime: 6h 52m 36.00s (Approximately 0.25 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  configFile: /home/grzegorz/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: /usr/bin/fuse-overlayfs is owned by fuse-overlayfs 1.13-1
      Version: |-
        fusermount3 version: 3.16.2
        fuse-overlayfs: version 1.13-dev
        FUSE library version 3.16.2
        using FUSE kernel interface version 7.38
  graphRoot: /home/grzegorz/.local/share/containers/storage
  graphRootAllocated: 983335399424
  graphRootUsed: 785074339840
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 27
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/grzegorz/.local/share/containers/storage/volumes
version:
  APIVersion: 4.8.2
  Built: 1702505246
  BuiltTime: Wed Dec 13 23:07:26 2023
  GitCommit: aa546902fa1a927b3d770528565627d1395b19f3-dirty
  GoVersion: go1.21.5
  Os: linux
  OsArch: linux/amd64
  Version: 4.8.2

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

Linux lenovo 6.6.7-arch1-1 #1 SMP PREEMPT_DYNAMIC Thu, 14 Dec 2023 03:45:42 +0000 x86_64 GNU/Linux

Additional information

--log-level debug

podman run --log-level debug -it --rm python
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called run.PersistentPreRunE(podman run --log-level debug -it --rm python) 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
INFO[0000] Using boltdb as database backend             
DEBU[0000] Initializing boltdb state at /home/grzegorz/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/grzegorz/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /home/grzegorz/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /home/grzegorz/.local/share/containers/storage/volumes 
DEBU[0000] Using transient store: false                 
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument 
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument 
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument 
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument 
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Setting parallel job count to 37             
DEBU[0000] Successfully loaded network podman: &{podman 2f259bab93aaaaa2542ba43ef33eb990d0999ee1b9924b557b7be53c0b7a1bb9 bridge cni-podman0 2021-08-21 23:41:52.76992324 +0200 CEST [{{{10.88.0.0 ffff0000}} 10.88.0.1 <nil>}] [] false false false [] map[] map[] map[driver:host-local]} 
DEBU[0000] Successfully loaded 1 networks               
DEBU[0000] Pulling image python (policy: missing)       
DEBU[0000] Looking up image "python" in local containers storage 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/00-shortnames.conf" 
DEBU[0000] Trying "docker.io/library/python:latest" ... 
DEBU[0000] parsed reference into "[overlay@/home/grzegorz/.local/share/containers/storage+/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mount_program=/usr/bin/fuse-overlayfs]@58a8f3dcd68a25102665617db6b9cc605dac7e5b84a874c456692513d12c990f" 
DEBU[0000] Found image "python" as "docker.io/library/python:latest" in local containers storage 
DEBU[0000] Found image "python" as "docker.io/library/python:latest" in local containers storage ([overlay@/home/grzegorz/.local/share/containers/storage+/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mount_program=/usr/bin/fuse-overlayfs]@58a8f3dcd68a25102665617db6b9cc605dac7e5b84a874c456692513d12c990f) 
DEBU[0000] exporting opaque data as blob "sha256:58a8f3dcd68a25102665617db6b9cc605dac7e5b84a874c456692513d12c990f" 
DEBU[0000] Looking up image "docker.io/library/python:latest" in local containers storage 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Trying "docker.io/library/python:latest" ... 
DEBU[0000] parsed reference into "[overlay@/home/grzegorz/.local/share/containers/storage+/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mount_program=/usr/bin/fuse-overlayfs]@58a8f3dcd68a25102665617db6b9cc605dac7e5b84a874c456692513d12c990f" 
DEBU[0000] Found image "docker.io/library/python:latest" as "docker.io/library/python:latest" in local containers storage 
DEBU[0000] Found image "docker.io/library/python:latest" as "docker.io/library/python:latest" in local containers storage ([overlay@/home/grzegorz/.local/share/containers/storage+/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mount_program=/usr/bin/fuse-overlayfs]@58a8f3dcd68a25102665617db6b9cc605dac7e5b84a874c456692513d12c990f) 
DEBU[0000] exporting opaque data as blob "sha256:58a8f3dcd68a25102665617db6b9cc605dac7e5b84a874c456692513d12c990f" 
DEBU[0000] Looking up image "python" in local containers storage 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Trying "docker.io/library/python:latest" ... 
DEBU[0000] parsed reference into "[overlay@/home/grzegorz/.local/share/containers/storage+/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mount_program=/usr/bin/fuse-overlayfs]@58a8f3dcd68a25102665617db6b9cc605dac7e5b84a874c456692513d12c990f" 
DEBU[0000] Found image "python" as "docker.io/library/python:latest" in local containers storage 
DEBU[0000] Found image "python" as "docker.io/library/python:latest" in local containers storage ([overlay@/home/grzegorz/.local/share/containers/storage+/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mount_program=/usr/bin/fuse-overlayfs]@58a8f3dcd68a25102665617db6b9cc605dac7e5b84a874c456692513d12c990f) 
DEBU[0000] exporting opaque data as blob "sha256:58a8f3dcd68a25102665617db6b9cc605dac7e5b84a874c456692513d12c990f" 
DEBU[0000] Inspecting image 58a8f3dcd68a25102665617db6b9cc605dac7e5b84a874c456692513d12c990f 
DEBU[0000] exporting opaque data as blob "sha256:58a8f3dcd68a25102665617db6b9cc605dac7e5b84a874c456692513d12c990f" 
DEBU[0000] exporting opaque data as blob "sha256:58a8f3dcd68a25102665617db6b9cc605dac7e5b84a874c456692513d12c990f" 
DEBU[0000] Inspecting image 58a8f3dcd68a25102665617db6b9cc605dac7e5b84a874c456692513d12c990f 
DEBU[0000] Inspecting image 58a8f3dcd68a25102665617db6b9cc605dac7e5b84a874c456692513d12c990f 
DEBU[0000] Inspecting image 58a8f3dcd68a25102665617db6b9cc605dac7e5b84a874c456692513d12c990f 
DEBU[0000] using systemd mode: false                    
DEBU[0000] No hostname set; container's hostname will default to runtime default 
DEBU[0000] Loading seccomp profile from "/etc/containers/seccomp.json" 
DEBU[0000] Allocated lock 1 for container ec388b61a77de24b5e0b51d83334f4f2c58c40790ae9adcd1d54b215b094be97 
DEBU[0000] parsed reference into "[overlay@/home/grzegorz/.local/share/containers/storage+/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mount_program=/usr/bin/fuse-overlayfs]@58a8f3dcd68a25102665617db6b9cc605dac7e5b84a874c456692513d12c990f" 
DEBU[0000] exporting opaque data as blob "sha256:58a8f3dcd68a25102665617db6b9cc605dac7e5b84a874c456692513d12c990f" 
DEBU[0000] Created container "ec388b61a77de24b5e0b51d83334f4f2c58c40790ae9adcd1d54b215b094be97" 
DEBU[0000] Container "ec388b61a77de24b5e0b51d83334f4f2c58c40790ae9adcd1d54b215b094be97" has work directory "/home/grzegorz/.local/share/containers/storage/overlay-containers/ec388b61a77de24b5e0b51d83334f4f2c58c40790ae9adcd1d54b215b094be97/userdata" 
DEBU[0000] Container "ec388b61a77de24b5e0b51d83334f4f2c58c40790ae9adcd1d54b215b094be97" has run directory "/run/user/1000/containers/overlay-containers/ec388b61a77de24b5e0b51d83334f4f2c58c40790ae9adcd1d54b215b094be97/userdata" 
DEBU[0000] Handling terminal attach                     
INFO[0000] Received shutdown.Stop(), terminating!        PID=142102
DEBU[0000] Enabling signal proxying                     
DEBU[0000] overlay: mount_data=lowerdir=/home/grzegorz/.local/share/containers/storage/overlay/l/WVF47XGFBBBXOQWWJN4TX4WXIX:/home/grzegorz/.local/share/containers/storage/overlay/l/GXAYI5TOMVCJN43KL73VANYOKL:/home/grzegorz/.local/share/containers/storage/overlay/l/XBKPN5K2QO7CTZG2FDK22QONJA:/home/grzegorz/.local/share/containers/storage/overlay/l/P3HGPH2J5HPFMWKMMYOEFDU273:/home/grzegorz/.local/share/containers/storage/overlay/l/JWDCIAELIC4SL6465KB6MPJIB2:/home/grzegorz/.local/share/containers/storage/overlay/l/E3BOPVAC3XUP7R6KCEFEMBV5IB:/home/grzegorz/.local/share/containers/storage/overlay/l/6I2DCCOUEPXECPUX4SUXHEYHI6:/home/grzegorz/.local/share/containers/storage/overlay/l/GG3KEYSOPLJIPTYXE4JRFCXJ4B,upperdir=/home/grzegorz/.local/share/containers/storage/overlay/f5e034fa6bf4d48130e4157e75a06dce58c00a61a7fb2e6bb909eec3fd4c0a94/diff,workdir=/home/grzegorz/.local/share/containers/storage/overlay/f5e034fa6bf4d48130e4157e75a06dce58c00a61a7fb2e6bb909eec3fd4c0a94/work,volatile 
DEBU[0000] Made network namespace at /run/user/1000/netns/netns-a54ef5bd-9056-d4dc-0d90-6fb0342c9615 for container ec388b61a77de24b5e0b51d83334f4f2c58c40790ae9adcd1d54b215b094be97 
DEBU[0000] slirp4netns command: /usr/bin/slirp4netns --disable-host-loopback --mtu=65520 --enable-sandbox --enable-seccomp --enable-ipv6 -c -r 3 -e 4 --netns-type=path /run/user/1000/netns/netns-a54ef5bd-9056-d4dc-0d90-6fb0342c9615 tap0 
DEBU[0000] Mounted container "ec388b61a77de24b5e0b51d83334f4f2c58c40790ae9adcd1d54b215b094be97" at "/home/grzegorz/.local/share/containers/storage/overlay/f5e034fa6bf4d48130e4157e75a06dce58c00a61a7fb2e6bb909eec3fd4c0a94/merged" 
DEBU[0000] Created root filesystem for container ec388b61a77de24b5e0b51d83334f4f2c58c40790ae9adcd1d54b215b094be97 at /home/grzegorz/.local/share/containers/storage/overlay/f5e034fa6bf4d48130e4157e75a06dce58c00a61a7fb2e6bb909eec3fd4c0a94/merged 
DEBU[0000] Skipping unrecognized mount in /etc/containers/mounts.conf: "# Configuration file for default mounts in containers (see man 5" 
DEBU[0000] Skipping unrecognized mount in /etc/containers/mounts.conf: "# containers-mounts.conf for further information)" 
DEBU[0000] Skipping unrecognized mount in /etc/containers/mounts.conf: "" 
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode subscription 
DEBU[0000] reading hooks from /usr/share/containers/oci/hooks.d 
DEBU[0000] added hook /usr/share/containers/oci/hooks.d/00-oci-nvidia-hook.json 
DEBU[0000] hook 00-oci-nvidia-hook.json matched; adding to stages [prestart] 
DEBU[0000] Workdir "/" resolved to host path "/home/grzegorz/.local/share/containers/storage/overlay/f5e034fa6bf4d48130e4157e75a06dce58c00a61a7fb2e6bb909eec3fd4c0a94/merged" 
DEBU[0000] Created OCI spec for container ec388b61a77de24b5e0b51d83334f4f2c58c40790ae9adcd1d54b215b094be97 at /home/grzegorz/.local/share/containers/storage/overlay-containers/ec388b61a77de24b5e0b51d83334f4f2c58c40790ae9adcd1d54b215b094be97/userdata/config.json 
DEBU[0000] /usr/bin/conmon messages will be logged to syslog 
DEBU[0000] running conmon: /usr/bin/conmon               args="[--api-version 1 -c ec388b61a77de24b5e0b51d83334f4f2c58c40790ae9adcd1d54b215b094be97 -u ec388b61a77de24b5e0b51d83334f4f2c58c40790ae9adcd1d54b215b094be97 -r /usr/bin/crun -b /home/grzegorz/.local/share/containers/storage/overlay-containers/ec388b61a77de24b5e0b51d83334f4f2c58c40790ae9adcd1d54b215b094be97/userdata -p /run/user/1000/containers/overlay-containers/ec388b61a77de24b5e0b51d83334f4f2c58c40790ae9adcd1d54b215b094be97/userdata/pidfile -n beautiful_hopper --exit-dir /run/user/1000/libpod/tmp/exits --full-attach -l journald --log-level debug --syslog -t --conmon-pidfile /run/user/1000/containers/overlay-containers/ec388b61a77de24b5e0b51d83334f4f2c58c40790ae9adcd1d54b215b094be97/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /home/grzegorz/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1000/containers --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg cgroupfs --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --network-config-dir --exit-command-arg  --exit-command-arg --network-backend --exit-command-arg cni --exit-command-arg --volumepath --exit-command-arg /home/grzegorz/.local/share/containers/storage/volumes --exit-command-arg --db-backend --exit-command-arg boltdb --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/usr/bin/fuse-overlayfs --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/usr/bin/fuse-overlayfs --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg --syslog --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg ec388b61a77de24b5e0b51d83334f4f2c58c40790ae9adcd1d54b215b094be97]"
DEBU[0000] Cleaning up container ec388b61a77de24b5e0b51d83334f4f2c58c40790ae9adcd1d54b215b094be97 
DEBU[0000] Tearing down network namespace at /run/user/1000/netns/netns-a54ef5bd-9056-d4dc-0d90-6fb0342c9615 for container ec388b61a77de24b5e0b51d83334f4f2c58c40790ae9adcd1d54b215b094be97 
DEBU[0000] Unmounted container "ec388b61a77de24b5e0b51d83334f4f2c58c40790ae9adcd1d54b215b094be97" 
DEBU[0000] Removing container ec388b61a77de24b5e0b51d83334f4f2c58c40790ae9adcd1d54b215b094be97 
DEBU[0000] Cleaning up container ec388b61a77de24b5e0b51d83334f4f2c58c40790ae9adcd1d54b215b094be97 
DEBU[0000] Network is already cleaned up, skipping...   
DEBU[0000] Container ec388b61a77de24b5e0b51d83334f4f2c58c40790ae9adcd1d54b215b094be97 storage is already unmounted, skipping... 
DEBU[0000] Removing all exec sessions for container ec388b61a77de24b5e0b51d83334f4f2c58c40790ae9adcd1d54b215b094be97 
DEBU[0000] Container ec388b61a77de24b5e0b51d83334f4f2c58c40790ae9adcd1d54b215b094be97 storage is already unmounted, skipping... 
DEBU[0000] ExitCode msg: "container create failed (no logs from conmon): conmon bytes \"\": readobjectstart: expect { or n, but found \x00, error found in #0 byte of ...||..., bigger context ...||..." 
Error: container create failed (no logs from conmon): conmon bytes "": readObjectStart: expect { or n, but found , error found in #0 byte of ...||..., bigger context ...||...
DEBU[0000] Shutting down engines
@grzegorzk grzegorzk added the kind/bug Categorizes issue or PR as related to a bug. label Dec 15, 2023
@grzegorzk
Copy link
Author

It seems podman fails on this step:

DEBU[0000] Handling terminal attach                     
INFO[0000] Received shutdown.Stop(), terminating!        PID=142102

@grzegorzk
Copy link
Author

grzegorzk commented Dec 15, 2023 •
edited
Loading

Running as root yields different error:

sudo podman run -it --rm python
Error: crun: realpath `3N�Z` failed: No such file or directory: OCI runtime attempted to invoke a command that was not found

Same as above, with --log-level debug

sudo podman run --log-level debug -it --rm python
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called run.PersistentPreRunE(podman run --log-level debug -it --rm python) 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
INFO[0000] Using boltdb as database backend             
DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db 
DEBU[0000] Overriding run root "/run/containers/storage" with "/var/run/containers/storage" from database 
DEBU[0000] Overriding tmp dir "/run/libpod" with "/var/run/libpod" from database 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/lib/containers/storage 
DEBU[0000] Using run root /var/run/containers/storage   
DEBU[0000] Using static dir /var/lib/containers/storage/libpod 
DEBU[0000] Using tmp dir /var/run/libpod                
DEBU[0000] Using volume path /var/lib/containers/storage/volumes 
DEBU[0000] Using transient store: false                 
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that metacopy is being used 
DEBU[0000] Cached value indicated that native-diff is not being used 
INFO[0000] Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=true 
DEBU[0000] Initializing event backend journald          
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument 
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument 
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument 
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument 
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument 
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Setting parallel job count to 37             
DEBU[0000] Successfully loaded network podman: &{podman 2f259bab93aaaaa2542ba43ef33eb990d0999ee1b9924b557b7be53c0b7a1bb9 bridge cni-podman0 2021-09-03 18:35:05.788440864 +0200 CEST [{{{10.88.0.0 ffff0000}} 10.88.0.1 <nil>}] [] false false false [] map[] map[] map[driver:host-local]} 
DEBU[0000] Successfully loaded 1 networks               
DEBU[0000] Pulling image python (policy: missing)       
DEBU[0000] Looking up image "python" in local containers storage 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/00-shortnames.conf" 
DEBU[0000] Trying "docker.io/library/python:latest" ... 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev]@58a8f3dcd68a25102665617db6b9cc605dac7e5b84a874c456692513d12c990f" 
DEBU[0000] Found image "python" as "docker.io/library/python:latest" in local containers storage 
DEBU[0000] Found image "python" as "docker.io/library/python:latest" in local containers storage ([overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev]@58a8f3dcd68a25102665617db6b9cc605dac7e5b84a874c456692513d12c990f) 
DEBU[0000] exporting opaque data as blob "sha256:58a8f3dcd68a25102665617db6b9cc605dac7e5b84a874c456692513d12c990f" 
DEBU[0000] Looking up image "docker.io/library/python:latest" in local containers storage 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Trying "docker.io/library/python:latest" ... 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev]@58a8f3dcd68a25102665617db6b9cc605dac7e5b84a874c456692513d12c990f" 
DEBU[0000] Found image "docker.io/library/python:latest" as "docker.io/library/python:latest" in local containers storage 
DEBU[0000] Found image "docker.io/library/python:latest" as "docker.io/library/python:latest" in local containers storage ([overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev]@58a8f3dcd68a25102665617db6b9cc605dac7e5b84a874c456692513d12c990f) 
DEBU[0000] exporting opaque data as blob "sha256:58a8f3dcd68a25102665617db6b9cc605dac7e5b84a874c456692513d12c990f" 
DEBU[0000] Looking up image "python" in local containers storage 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Trying "docker.io/library/python:latest" ... 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev]@58a8f3dcd68a25102665617db6b9cc605dac7e5b84a874c456692513d12c990f" 
DEBU[0000] Found image "python" as "docker.io/library/python:latest" in local containers storage 
DEBU[0000] Found image "python" as "docker.io/library/python:latest" in local containers storage ([overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev]@58a8f3dcd68a25102665617db6b9cc605dac7e5b84a874c456692513d12c990f) 
DEBU[0000] exporting opaque data as blob "sha256:58a8f3dcd68a25102665617db6b9cc605dac7e5b84a874c456692513d12c990f" 
DEBU[0000] Inspecting image 58a8f3dcd68a25102665617db6b9cc605dac7e5b84a874c456692513d12c990f 
DEBU[0000] exporting opaque data as blob "sha256:58a8f3dcd68a25102665617db6b9cc605dac7e5b84a874c456692513d12c990f" 
DEBU[0000] exporting opaque data as blob "sha256:58a8f3dcd68a25102665617db6b9cc605dac7e5b84a874c456692513d12c990f" 
DEBU[0000] Inspecting image 58a8f3dcd68a25102665617db6b9cc605dac7e5b84a874c456692513d12c990f 
DEBU[0000] Inspecting image 58a8f3dcd68a25102665617db6b9cc605dac7e5b84a874c456692513d12c990f 
DEBU[0000] Inspecting image 58a8f3dcd68a25102665617db6b9cc605dac7e5b84a874c456692513d12c990f 
DEBU[0000] using systemd mode: false                    
DEBU[0000] No hostname set; container's hostname will default to runtime default 
DEBU[0000] Loading seccomp profile from "/etc/containers/seccomp.json" 
DEBU[0000] Allocated lock 0 for container 7891b72dfefd81e45e0d1a5cec66c39ecfa19f96e3a381eb106c5fdcf96bb9a9 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev]@58a8f3dcd68a25102665617db6b9cc605dac7e5b84a874c456692513d12c990f" 
DEBU[0000] exporting opaque data as blob "sha256:58a8f3dcd68a25102665617db6b9cc605dac7e5b84a874c456692513d12c990f" 
DEBU[0000] Cached value indicated that idmapped mounts for overlay are supported 
DEBU[0000] Created container "7891b72dfefd81e45e0d1a5cec66c39ecfa19f96e3a381eb106c5fdcf96bb9a9" 
DEBU[0000] Container "7891b72dfefd81e45e0d1a5cec66c39ecfa19f96e3a381eb106c5fdcf96bb9a9" has work directory "/var/lib/containers/storage/overlay-containers/7891b72dfefd81e45e0d1a5cec66c39ecfa19f96e3a381eb106c5fdcf96bb9a9/userdata" 
DEBU[0000] Container "7891b72dfefd81e45e0d1a5cec66c39ecfa19f96e3a381eb106c5fdcf96bb9a9" has run directory "/var/run/containers/storage/overlay-containers/7891b72dfefd81e45e0d1a5cec66c39ecfa19f96e3a381eb106c5fdcf96bb9a9/userdata" 
DEBU[0000] Handling terminal attach                     
INFO[0000] Received shutdown.Stop(), terminating!        PID=143651
DEBU[0000] Enabling signal proxying                     
DEBU[0000] Cached value indicated that volatile is being used 
DEBU[0000] overlay: mount_data=lowerdir=/var/lib/containers/storage/overlay/l/J7F3G7B7USXTQKMXDPEU7HYJU2:/var/lib/containers/storage/overlay/l/JCKQL4MTUUJ6BB47WEDE6BHAQO:/var/lib/containers/storage/overlay/l/4CQZH6OB6T6BEQLGJRWAL3WUSN:/var/lib/containers/storage/overlay/l/C4KJ5N37HFF7NBH4LIBRTI4YFP:/var/lib/containers/storage/overlay/l/DRVY3FPXZJMUG7N6YANJP7RIKC:/var/lib/containers/storage/overlay/l/PLPF4KGWXMTR72YF6GA2KL765M:/var/lib/containers/storage/overlay/l/GT7SPVN6J4TQF3EL4M7SQLNHCK:/var/lib/containers/storage/overlay/l/QLV2OKL7ILLJNJ4T6XKF7PE33O,upperdir=/var/lib/containers/storage/overlay/a6c53b4363a92dc913c97a0451bd72eacd2af71c0de39d9a333c584b44c28052/diff,workdir=/var/lib/containers/storage/overlay/a6c53b4363a92dc913c97a0451bd72eacd2af71c0de39d9a333c584b44c28052/work,nodev,volatile 
DEBU[0000] Made network namespace at /run/netns/netns-29dafb0f-336f-8243-b698-4f74a8fb4cf2 for container 7891b72dfefd81e45e0d1a5cec66c39ecfa19f96e3a381eb106c5fdcf96bb9a9 
DEBU[0000] Mounted container "7891b72dfefd81e45e0d1a5cec66c39ecfa19f96e3a381eb106c5fdcf96bb9a9" at "/var/lib/containers/storage/overlay/a6c53b4363a92dc913c97a0451bd72eacd2af71c0de39d9a333c584b44c28052/merged" 
DEBU[0000] Created root filesystem for container 7891b72dfefd81e45e0d1a5cec66c39ecfa19f96e3a381eb106c5fdcf96bb9a9 at /var/lib/containers/storage/overlay/a6c53b4363a92dc913c97a0451bd72eacd2af71c0de39d9a333c584b44c28052/merged 
DEBU[0000] cni result for container 7891b72dfefd81e45e0d1a5cec66c39ecfa19f96e3a381eb106c5fdcf96bb9a9 network podman: &{0.4.0 [{Name:cni-podman0 Mac:aa:cd:e4:cc:8d:88 Sandbox:} {Name:veth04d8c36b Mac:32:80:c0:c7:9c:1a Sandbox:} {Name:eth0 Mac:ea:5e:39:d2:c5:38 Sandbox:/run/netns/netns-29dafb0f-336f-8243-b698-4f74a8fb4cf2}] [{Version:4 Interface:0xc00047b858 Address:{IP:10.88.0.4 Mask:ffff0000} Gateway:10.88.0.1}] [{Dst:{IP:0.0.0.0 Mask:00000000} GW:<nil>}] {[]  [] []}} 
DEBU[0000] Skipping unrecognized mount in /etc/containers/mounts.conf: "# Configuration file for default mounts in containers (see man 5" 
DEBU[0000] Skipping unrecognized mount in /etc/containers/mounts.conf: "# containers-mounts.conf for further information)" 
DEBU[0000] Skipping unrecognized mount in /etc/containers/mounts.conf: "" 
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode subscription 
DEBU[0000] Setting Cgroups for container 7891b72dfefd81e45e0d1a5cec66c39ecfa19f96e3a381eb106c5fdcf96bb9a9 to machine.slice:libpod:7891b72dfefd81e45e0d1a5cec66c39ecfa19f96e3a381eb106c5fdcf96bb9a9 
DEBU[0000] reading hooks from /usr/share/containers/oci/hooks.d 
DEBU[0000] added hook /usr/share/containers/oci/hooks.d/00-oci-nvidia-hook.json 
DEBU[0000] hook 00-oci-nvidia-hook.json matched; adding to stages [prestart] 
DEBU[0000] Workdir "/" resolved to host path "/var/lib/containers/storage/overlay/a6c53b4363a92dc913c97a0451bd72eacd2af71c0de39d9a333c584b44c28052/merged" 
DEBU[0000] Created OCI spec for container 7891b72dfefd81e45e0d1a5cec66c39ecfa19f96e3a381eb106c5fdcf96bb9a9 at /var/lib/containers/storage/overlay-containers/7891b72dfefd81e45e0d1a5cec66c39ecfa19f96e3a381eb106c5fdcf96bb9a9/userdata/config.json 
DEBU[0000] /usr/bin/conmon messages will be logged to syslog 
DEBU[0000] running conmon: /usr/bin/conmon               args="[--api-version 1 -c 7891b72dfefd81e45e0d1a5cec66c39ecfa19f96e3a381eb106c5fdcf96bb9a9 -u 7891b72dfefd81e45e0d1a5cec66c39ecfa19f96e3a381eb106c5fdcf96bb9a9 -r /usr/bin/crun -b /var/lib/containers/storage/overlay-containers/7891b72dfefd81e45e0d1a5cec66c39ecfa19f96e3a381eb106c5fdcf96bb9a9/userdata -p /var/run/containers/storage/overlay-containers/7891b72dfefd81e45e0d1a5cec66c39ecfa19f96e3a381eb106c5fdcf96bb9a9/userdata/pidfile -n practical_gould --exit-dir /var/run/libpod/exits --full-attach -s -l journald --log-level debug --syslog -t --conmon-pidfile /var/run/containers/storage/overlay-containers/7891b72dfefd81e45e0d1a5cec66c39ecfa19f96e3a381eb106c5fdcf96bb9a9/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /var/run/containers/storage --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /var/run/libpod --exit-command-arg --network-config-dir --exit-command-arg  --exit-command-arg --network-backend --exit-command-arg cni --exit-command-arg --volumepath --exit-command-arg /var/lib/containers/storage/volumes --exit-command-arg --db-backend --exit-command-arg boltdb --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mountopt=nodev --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg --syslog --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg 7891b72dfefd81e45e0d1a5cec66c39ecfa19f96e3a381eb106c5fdcf96bb9a9]"
INFO[0000] Running conmon under slice machine.slice and unitName libpod-conmon-7891b72dfefd81e45e0d1a5cec66c39ecfa19f96e3a381eb106c5fdcf96bb9a9.scope 
DEBU[0000] Received: -1                                 
DEBU[0000] Cleaning up container 7891b72dfefd81e45e0d1a5cec66c39ecfa19f96e3a381eb106c5fdcf96bb9a9 
DEBU[0000] Tearing down network namespace at /run/netns/netns-29dafb0f-336f-8243-b698-4f74a8fb4cf2 for container 7891b72dfefd81e45e0d1a5cec66c39ecfa19f96e3a381eb106c5fdcf96bb9a9 
DEBU[0000] Unmounted container "7891b72dfefd81e45e0d1a5cec66c39ecfa19f96e3a381eb106c5fdcf96bb9a9" 
DEBU[0000] Removing container 7891b72dfefd81e45e0d1a5cec66c39ecfa19f96e3a381eb106c5fdcf96bb9a9 
DEBU[0000] Cleaning up container 7891b72dfefd81e45e0d1a5cec66c39ecfa19f96e3a381eb106c5fdcf96bb9a9 
DEBU[0000] Network is already cleaned up, skipping...   
DEBU[0000] Container 7891b72dfefd81e45e0d1a5cec66c39ecfa19f96e3a381eb106c5fdcf96bb9a9 storage is already unmounted, skipping... 
DEBU[0000] Removing all exec sessions for container 7891b72dfefd81e45e0d1a5cec66c39ecfa19f96e3a381eb106c5fdcf96bb9a9 
DEBU[0000] Container 7891b72dfefd81e45e0d1a5cec66c39ecfa19f96e3a381eb106c5fdcf96bb9a9 storage is already unmounted, skipping... 
DEBU[0000] ExitCode msg: "crun: realpath `\x05�;c\x05` failed: no such file or directory: oci runtime attempted to invoke a command that was not found" 
Error: crun: realpath `�;c` failed: No such file or directory: OCI runtime attempted to invoke a command that was not found
DEBU[0000] Shutting down engines

@Luap99
Copy link
Member

Luap99 commented Dec 15, 2023

Duplicate of #21024

@Luap99 Luap99 marked this as a duplicate of #21024 Dec 15, 2023
@Luap99 Luap99 closed this as not planned Won't fix, can't repro, duplicate, stale Dec 15, 2023
@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Mar 15, 2024
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Mar 15, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@grzegorzk @Luap99