Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"Error: failed to start container fedora-toolbox-32" on Fedora Silverblue #500

Closed
rugk opened this issue Jul 12, 2020 · 17 comments
Closed
Assignees
Labels
1. Bug Something isn't working

Comments

@rugk
Copy link

rugk commented Jul 12, 2020

Fedora Silverblue 32, 32.20200712.0 (2020-07-12T00:42:58Z)
$USER = rugk

logs

$ toolbox enter
#Error: failed to start container fedora-toolbox-32
$ podman logs fedora-toolbox-32
toolbox: running as real user ID 0
toolbox: resolved absolute path for /usr/bin/toolbox to /usr/bin/toolbox
toolbox: TOOLBOX_PATH is /usr/bin/toolbox
toolbox: XDG_RUNTIME_DIR is unset
toolbox: XDG_RUNTIME_DIR set to /run/user/1000
toolbox: creating /run/.toolboxenv
toolbox: redirecting /etc/host.conf to /run/host/etc/host.conf
toolbox: redirecting /etc/hosts to /run/host/etc/hosts
toolbox: redirecting /etc/resolv.conf to /run/host/etc/resolv.conf
toolbox: binding /etc/machine-id to /run/host/etc/machine-id
toolbox: creating /run/systemd/journal
toolbox: binding /run/systemd/journal to /run/host/run/systemd/journal
toolbox: creating /sys/fs/selinux
toolbox: binding /sys/fs/selinux to /usr/share/empty
toolbox: creating /var/lib/flatpak
toolbox: binding /var/lib/flatpak to /run/host/var/lib/flatpak
toolbox: creating /var/log/journal
toolbox: binding /var/log/journal to /run/host/var/log/journal
toolbox: creating /var/mnt
toolbox: binding /var/mnt to /run/host/var/mnt
toolbox: redirecting /etc/localtime to /run/host/monitor/localtime
toolbox: redirecting /etc/timezone to /run/host/monitor/timezone
toolbox: making /media a symbolic link to /run/media
toolbox: making /mnt a symbolic link to /var/mnt
id: 'rugk': no such user
toolbox: making /home a symlink
toolbox: adding user rugk with UID 1000
useradd: Warning: missing or non-executable shell '/bin/zsh'
toolbox: removing password for user rugk
passwd: Note: deleting a password also unlocks the password.
toolbox: removing password for user root
passwd: Note: deleting a password also unlocks the password.
toolbox: setting KCM as the default Kerberos credential cache
toolbox: finished initializing container
toolbox: going to sleep

$ podman --log-level debug start fedora-toolbox-32
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called start.PersistentPreRunE(podman --log-level debug start fedora-toolbox-32) 
DEBU[0000] Ignoring libpod.conf EventsLogger setting "/var/home/rugk/.config/containers/containers.conf". Use "journald" if you want to change this setting and remove libpod.conf files. 
DEBU[0000] Reading configuration file "/usr/share/containers/containers.conf" 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf": &{{[] [] containers-default-0.14.4 [] private enabled [CAP_AUDIT_WRITE CAP_CHOWN CAP_DAC_OVERRIDE CAP_FOWNER CAP_FSETID CAP_KILL CAP_MKNOD CAP_NET_BIND_SERVICE CAP_NET_RAW CAP_SETFCAP CAP_SETGID CAP_SETPCAP CAP_SETUID CAP_SYS_CHROOT] [] []  [] [] [] true [PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] false false false  private k8s-file -1 slirp4netns false 2048 private /usr/share/containers/seccomp.json 65536k private host 65536} {true systemd [PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] [/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/conmon /usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon] ctrl-p,ctrl-q true /run/user/1000/libpod/tmp/events/events.log file [/usr/share/containers/oci/hooks.d] docker:// /pause k8s.gcr.io/pause:3.2 /usr/libexec/podman/catatonit shm   false 2048 /usr/bin/crun map[crun:[/usr/bin/crun /usr/sbin/crun /usr/local/bin/crun /usr/local/sbin/crun /sbin/crun /bin/crun /run/current-system/sw/bin/crun] kata:[/usr/bin/kata-runtime /usr/sbin/kata-runtime /usr/local/bin/kata-runtime /usr/local/sbin/kata-runtime /sbin/kata-runtime /bin/kata-runtime /usr/bin/kata-qemu /usr/bin/kata-fc] runc:[/usr/bin/runc /usr/sbin/runc /usr/local/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc]] missing false   [] [crun runc] [crun] [kata kata-runtime kata-qemu kata-fc] {false false false false false false} /etc/containers/policy.json false 3 /var/home/rugk/.local/share/containers/storage/libpod 10 /run/user/1000/libpod/tmp /var/home/rugk/.local/share/containers/storage/volumes} {[/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin] podman /etc/cni/net.d/}} 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/home/rugk/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/rugk/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /var/home/rugk/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /var/home/rugk/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Initializing event backend file              
DEBU[0000] using runtime "/usr/bin/runc"                
DEBU[0000] using runtime "/usr/bin/crun"                
WARN[0000] Error initializing configured OCI runtime kata: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] using runtime "/usr/bin/crun"                
INFO[0000] Setting parallel job count to 49             
DEBU[0000] overlay: mount_data=lowerdir=/var/home/rugk/.local/share/containers/storage/overlay/l/HHFMQSI45LPHONFF5SO75XX7BS:/var/home/rugk/.local/share/containers/storage/overlay/l/RYRUJGFXQY6SCM2AR2OXNWYPQP,upperdir=/var/home/rugk/.local/share/containers/storage/overlay/7d5b61e2bf03f015c6e164a34926fa52ab94518850fe3e6a0204b6fb59ef2b35/diff,workdir=/var/home/rugk/.local/share/containers/storage/overlay/7d5b61e2bf03f015c6e164a34926fa52ab94518850fe3e6a0204b6fb59ef2b35/work,context="system_u:object_r:container_file_t:s0:c539,c801" 
DEBU[0000] mounted container "462c77582f51221cb9dc8c98f800ec0ec99842a3147aeb6325142dee18349e9d" at "/var/home/rugk/.local/share/containers/storage/overlay/7d5b61e2bf03f015c6e164a34926fa52ab94518850fe3e6a0204b6fb59ef2b35/merged" 
DEBU[0000] Created root filesystem for container 462c77582f51221cb9dc8c98f800ec0ec99842a3147aeb6325142dee18349e9d at /var/home/rugk/.local/share/containers/storage/overlay/7d5b61e2bf03f015c6e164a34926fa52ab94518850fe3e6a0204b6fb59ef2b35/merged 
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode secret 
DEBU[0000] Setting CGroups for container 462c77582f51221cb9dc8c98f800ec0ec99842a3147aeb6325142dee18349e9d to user.slice:libpod:462c77582f51221cb9dc8c98f800ec0ec99842a3147aeb6325142dee18349e9d 
DEBU[0000] set root propagation to "rslave"             
DEBU[0000] reading hooks from /usr/share/containers/oci/hooks.d 
DEBU[0000] Created OCI spec for container 462c77582f51221cb9dc8c98f800ec0ec99842a3147aeb6325142dee18349e9d at /var/home/rugk/.local/share/containers/storage/overlay-containers/462c77582f51221cb9dc8c98f800ec0ec99842a3147aeb6325142dee18349e9d/userdata/config.json 
DEBU[0000] /usr/bin/conmon messages will be logged to syslog 
DEBU[0000] running conmon: /usr/bin/conmon               args="[--api-version 1 -c 462c77582f51221cb9dc8c98f800ec0ec99842a3147aeb6325142dee18349e9d -u 462c77582f51221cb9dc8c98f800ec0ec99842a3147aeb6325142dee18349e9d -r /usr/bin/crun -b /var/home/rugk/.local/share/containers/storage/overlay-containers/462c77582f51221cb9dc8c98f800ec0ec99842a3147aeb6325142dee18349e9d/userdata -p /run/user/1000/containers/overlay-containers/462c77582f51221cb9dc8c98f800ec0ec99842a3147aeb6325142dee18349e9d/userdata/pidfile -n fedora-toolbox-32 --exit-dir /run/user/1000/libpod/tmp/exits --socket-dir-path /run/user/1000/libpod/tmp/socket -s -l k8s-file:/var/home/rugk/.local/share/containers/storage/overlay-containers/462c77582f51221cb9dc8c98f800ec0ec99842a3147aeb6325142dee18349e9d/userdata/ctr.log --log-level debug --syslog --conmon-pidfile /run/user/1000/containers/overlay-containers/462c77582f51221cb9dc8c98f800ec0ec99842a3147aeb6325142dee18349e9d/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/home/rugk/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1000/containers --exit-command-arg --log-level --exit-command-arg error --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --runtime --exit-command-arg /usr/bin/crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/usr/bin/fuse-overlayfs --exit-command-arg --events-backend --exit-command-arg file --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 462c77582f51221cb9dc8c98f800ec0ec99842a3147aeb6325142dee18349e9d]"
[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

DEBU[0000] Received: -1                                 
DEBU[0000] Cleaning up container 462c77582f51221cb9dc8c98f800ec0ec99842a3147aeb6325142dee18349e9d 
DEBU[0000] Network is already cleaned up, skipping...   
DEBU[0000] unmounted container "462c77582f51221cb9dc8c98f800ec0ec99842a3147aeb6325142dee18349e9d" 
Error: unable to start container "462c77582f51221cb9dc8c98f800ec0ec99842a3147aeb6325142dee18349e9d": setrlimit `RLIMIT_NPROC`: Operation not permitted: OCI runtime permission denied error

podman container list lists no container.

@rugk rugk changed the title Error: failed to start container fedora-toolbox-32 "Error: failed to start container fedora-toolbox-32" on Fedora Silverblue Jul 12, 2020
@rugk
Copy link
Author

rugk commented Jul 12, 2020

Repulling comntainer…

$ podman run --it --rm registry.fedoraproject.org/fedora:32 /bin/bash
Error: unknown flag: --it
$ podman run --rm registry.fedoraproject.org/fedora:32 /bin/bash 
Trying to pull registry.fedoraproject.org/fedora:32...
Getting image source signatures
Copying blob dd9f43919ba0 done  
Copying config 00ff39a8bf done  
Writing manifest to image destination
Storing signatures

…did not work either.
I.e. toolbox enter still does not work.

@HarryMichal
Copy link
Member

Hey @rugk! It seems you're still running the Shell Toolbox (v0.0.18) even though your current deployment of Silverblue should already contain v0.0.92. Can you, please, try to upgrade your machine again and try to use the new Toolbox??

@HarryMichal HarryMichal self-assigned this Jul 13, 2020
@HarryMichal HarryMichal added the 1. Bug Something isn't working label Jul 13, 2020
@rugk
Copy link
Author

rugk commented Jul 13, 2020

toolbox --version returns 0.0.92.
Does not work, though. (still the same deployment AFAIK)

@HarryMichal
Copy link
Member

Hmm, odd. v0.0.18 prepends toolbox: to every log message, while the newer version uses a special logging package that uses a different format.

I have the same deployment and a newly created container is running alright.

According to the output of podman logs fedora-toolbox-32 the container uses the old Toolbox. Can you show me the output of podman inspect -t container fedora-toolbox-32??

@rugk
Copy link
Author

rugk commented Jul 13, 2020

Without knowing anything about the system, maybe the logs are still from an old toolbox version and it did not write any new logs?

podman inspect -t container fedora-toolbox-32
https://gist.github.com/rugk/6bd2a15378f9811e8db9cb0f5c964e96

IIRC I created that container a few months ago, not that long ago.

@HarryMichal
Copy link
Member

Without knowing anything about the system, maybe the logs are still from an old toolbox version and it did not write any new logs?

Good point! That actually may be the reason.

So, this happens with an old container... Can the issue be reproduced with a new container??

@mMerlin
Copy link

mMerlin commented Jul 14, 2020

I have the same issue with Fedora 31 Workstation
5.7.7-100.fc31.x86_64
Access to all of my existing toolbox containers are failing with toolbox: failed to start container «containerName»

tools --version says that is not a recognized option here. This was working yesterday. System changes since then was only dnf update, which was:

Packages Altered:
    Upgrade  librados2-2:14.2.10-1.fc31.x86_64 @updates
    Upgraded librados2-2:14.2.9-2.fc31.x86_64  @@System
    Upgrade  librbd1-2:14.2.10-1.fc31.x86_64   @updates
    Upgraded librbd1-2:14.2.9-2.fc31.x86_64    @@System

In case there was no reboot yesterday, the previous update did:

Packages Altered:
    Upgrade  LibRaw-0.19.5-3.fc31.x86_64                 @updates
    Upgraded LibRaw-0.19.5-1.fc31.x86_64                 @@System
    Upgrade  dkms-2.8.2-1.fc31.noarch                    @updates
    Upgraded dkms-2.8.1-4.20200214git5ca628c.fc31.noarch @@System

I could supply the output of podman inspect, but since I have 8 separate containers failing the same way, all at the same time, I doubt it is a container specific problem. Unless a toolbox run --container «name» sudo dnf -y upgrade triggered the problem. That was run yesterday on all of the containers. toolsbox list shows status "Exited (143) 10 hours ago", which is when the host system was shut down.

EDIT: So far, a newly created container is working correctly

@mMerlin
Copy link

mMerlin commented Jul 14, 2020

After building up a new container with the same (very close) configuration/content as one of the failing cases, I did podman inspect on each of them, and compared the outputs. The interesting differences are:

old failing                                                               new working
[                                                                         [
    {                                                                         {
        "State": {                                                                "State": {
            "OciVersion": "1.0.1-dev",                                   |            "OciVersion": "1.0.2-dev",
        },                                                                        },
        "MountLabel": "system_u:object_r:container_file_t:s0:c26,c243",  |        "MountLabel": "system_u:object_r:container_file_t:s0:c361,c604",
        "Config": {                                                               "Config": {
            "Env": [                                                                  "Env": [
                "container=oci",                                         |                "container=podman",
            ],                                                                        ],
        },                                                                         },
        "HostConfig": {                                                            "HostConfig": {
            "IpcMode": "host",                                           |             "IpcMode": "private",
            "MemorySwappiness": -1,                                      |             "MemorySwappiness": 0,
            "Ulimits": [                                                 |             "Ulimits": [],
                {                                                        <
                    "Name": "RLIMIT_NOFILE",                             <
                    "Soft": 524288,                                      <
                    "Hard": 524288                                       <
                },                                                       <
                {                                                        <
                    "Name": "RLIMIT_NPROC",                              <
                    "Soft": 63377,                                       <
                    "Hard": 63377                                        <
                }                                                        <
            ],                                                           <
        }                                                                          }
    }                                                                          }
]                                                                         ]

@rugk
Copy link
Author

rugk commented Jul 14, 2020

tools --version says that is not a recognized option here.

Obviously yes, because your command is wrong… 🙃 😉

@mMerlin
Copy link

mMerlin commented Jul 14, 2020

:)

% toolbox --version
toolbox: unrecognized option '--version'
Try 'toolbox --help' for more information.

Doesn't work either
:(

@HarryMichal
Copy link
Member

Thank you @mMerlin for all the information! It is very helpful! With this I know, this is caused by Podman itself. I found some issues in Podman's bug tracker: containers/podman#6857, containers/podman#6389.

Based on the changelog entry for Podman v2.0.2, the issue should be already fixed.

* Fixed a bug where NPROC and NOFILE rlimits could be improperly set for rootless Podman containers, causing them to fail to start.

Can you confirm that the issue is resolved or is it still present?

:)

% toolbox --version
toolbox: unrecognized option '--version'
Try 'toolbox --help' for more information.

Doesn't work either
:(

On Fedora 31 is still Toolbox v0.0.18 which does not have the option --version available. Instead, you can check the version with rpm -q toolbox.

@mMerlin
Copy link

mMerlin commented Jul 14, 2020

Apparently not fixed with versions I have available. Starting from:

% podman --version
podman version 2.0.1
% rpm -q toolbox
toolbox-0.0.18-2.fc31.noarch
% sudo dnf upgrade podman --enablerepo=updates-testing --best
% dnf history info 196
…
Command Line   : upgrade podman --enablerepo=updates-testing --best
Comment        :
Packages Altered:
    Upgrade  podman-2:2.0.2-1.fc31.x86_64         @updates-testing
    Upgraded podman-2:2.0.1-1.fc31.x86_64         @@System
    Upgrade  podman-docker-2:2.0.2-1.fc31.noarch  @updates-testing
    Upgraded podman-docker-2:2.0.1-1.fc31.noarch  @@System
    Upgrade  podman-plugins-2:2.0.2-1.fc31.x86_64 @updates-testing
    Upgraded podman-plugins-2:2.0.1-1.fc31.x86_64 @@System
% podman --version
podman version 2.0.2

With a system reboot for good measure, the old containers are still failing to start. No change in the symptoms.

@HarryMichal
Copy link
Member

Looks like on F31 the update did not fix the issue according to Bodhi: https://bodhi.fedoraproject.org/updates/FEDORA-2020-c510ceb7db. I encourage you to comment on the update and give it a negative karma.

@rugk, I believe the same could be done for the F32 update: https://bodhi.fedoraproject.org/updates/FEDORA-2020-19924b556e.

Apart from this, I'd possibly comment on one of the mentioned issue tickets to make it known that the bug has not been fixed.

@HarryMichal
Copy link
Member

I added a reproducer to the upstream issue.

@debarshiray
Copy link
Member

So it seems like we should sort it out in Podman, right?

Duplicate of containers/podman#6389 and containers/podman#6857

@biji
Copy link

biji commented Sep 21, 2020

Sorry how to fix this? Tried dnf downgrade podman but still cant enter, fedora-toolbox-32

Thanks

@biji
Copy link

biji commented Mar 4, 2021

#687

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
1. Bug Something isn't working
Projects
None yet
Development

No branches or pull requests

5 participants