Skip to content
This repository has been archived by the owner on Mar 9, 2022. It is now read-only.

Systemctl start containerd failed on RHEL #852

Closed
hmtai opened this issue Jul 18, 2018 · 3 comments
Closed

Systemctl start containerd failed on RHEL #852

hmtai opened this issue Jul 18, 2018 · 3 comments
Labels
kind/question

Comments

@hmtai
Copy link

hmtai commented Jul 18, 2018 •
edited
Loading

@Random-Liu When I install containerd through cri-containerd-1.1.0.linux-amd64.tar.gzpackage,contaienrd can not start.

# tar -C / -zxf cri-containerd-1.1.0.linux-amd64.tar.gz
# systemctl daemon-reload
# systemctl start containerd
# crictl info
bash: crictl: command not found

# journalctl -u containerd -f
-- Logs begin at Wed 2018-07-18 02:26:22 EDT. --
Jul 18 02:46:19 cri-rhel-1.novalocal containerd[12006]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x60 pc=0x55f2ee8ef54e]
Jul 18 02:46:19 cri-rhel-1.novalocal containerd[12006]: goroutine 63 [running]:
Jul 18 02:46:19 cri-rhel-1.novalocal containerd[12006]: github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server.(*snapshotsSyncer).sync(0xc4204944e0, 0x55f2ef14f0e0, 0xc4204922d0)
Jul 18 02:46:19 cri-rhel-1.novalocal containerd[12006]: /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/snapshots.go:79 +0x19e
Jul 18 02:46:19 cri-rhel-1.novalocal containerd[12006]: github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server.(*snapshotsSyncer).start.func1(0xc4204922d0, 0xc4204944e0)
Jul 18 02:46:19 cri-rhel-1.novalocal containerd[12006]: /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/snapshots.go:62 +0x77
Jul 18 02:46:19 cri-rhel-1.novalocal containerd[12006]: created by github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server.(*snapshotsSyncer).start
Jul 18 02:46:19 cri-rhel-1.novalocal containerd[12006]: /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/snapshots.go:57 +0x5d
Jul 18 02:46:19 cri-rhel-1.novalocal systemd[1]: Unit containerd.service entered failed state.
Jul 18 02:46:19 cri-rhel-1.novalocal systemd[1]: containerd.service failed.
Jul 18 02:46:25 cri-rhel-1.novalocal systemd[1]: containerd.service holdoff time over, scheduling restart.
Jul 18 02:46:25 cri-rhel-1.novalocal systemd[1]: Starting containerd container runtime...
Jul 18 02:46:25 cri-rhel-1.novalocal systemd[1]: Started containerd container runtime.
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=info msg="starting containerd" revision=209a7fc3e4a32ef71a8c7b50c68fc8398415badf version=v1.1.0
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=info msg="loading plugin "io.containerd.content.v1.content"..." type=io.containerd.content.v1
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=info msg="loading plugin "io.containerd.snapshotter.v1.btrfs"..." type=io.containerd.snapshotter.v1
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.btrfs" error="path /var/lib/containerd/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter"
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=info msg="loading plugin "io.containerd.snapshotter.v1.aufs"..." type=io.containerd.snapshotter.v1
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.aufs" error="modprobe aufs failed: "modprobe: FATAL: Module aufs not found.\n": exit status 1"
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=info msg="loading plugin "io.containerd.snapshotter.v1.native"..." type=io.containerd.snapshotter.v1
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=info msg="loading plugin "io.containerd.snapshotter.v1.overlayfs"..." type=io.containerd.snapshotter.v1
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.overlayfs" error="/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs does not support d_type. If the backing filesystem is xfs, please reformat with ftype=1 to enable d_type support"
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=info msg="loading plugin "io.containerd.snapshotter.v1.zfs"..." type=io.containerd.snapshotter.v1
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.zfs" error="path /var/lib/containerd/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter"
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=info msg="loading plugin "io.containerd.metadata.v1.bolt"..." type=io.containerd.metadata.v1
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=warning msg="could not use snapshotter overlayfs in metadata plugin" error="/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs does not support d_type. If the backing filesystem is xfs, please reformat with ftype=1 to enable d_type support"
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=warning msg="could not use snapshotter zfs in metadata plugin" error="path /var/lib/containerd/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter"
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=warning msg="could not use snapshotter btrfs in metadata plugin" error="path /var/lib/containerd/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter"
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=warning msg="could not use snapshotter aufs in metadata plugin" error="modprobe aufs failed: "modprobe: FATAL: Module aufs not found.\n": exit status 1"
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=info msg="loading plugin "io.containerd.differ.v1.walking"..." type=io.containerd.differ.v1
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=info msg="loading plugin "io.containerd.gc.v1.scheduler"..." type=io.containerd.gc.v1
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=info msg="loading plugin "io.containerd.service.v1.containers-service"..." type=io.containerd.service.v1
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=info msg="loading plugin "io.containerd.service.v1.content-service"..." type=io.containerd.service.v1
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=info msg="loading plugin "io.containerd.service.v1.diff-service"..." type=io.containerd.service.v1
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=info msg="loading plugin "io.containerd.service.v1.images-service"..." type=io.containerd.service.v1
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=info msg="loading plugin "io.containerd.service.v1.leases-service"..." type=io.containerd.service.v1
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=info msg="loading plugin "io.containerd.service.v1.namespaces-service"..." type=io.containerd.service.v1
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=info msg="loading plugin "io.containerd.service.v1.snapshots-service"..." type=io.containerd.service.v1
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=info msg="loading plugin "io.containerd.monitor.v1.cgroups"..." type=io.containerd.monitor.v1
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=info msg="loading plugin "io.containerd.runtime.v1.linux"..." type=io.containerd.runtime.v1
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=info msg="loading plugin "io.containerd.service.v1.tasks-service"..." type=io.containerd.service.v1
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=info msg="loading plugin "io.containerd.grpc.v1.containers"..." type=io.containerd.grpc.v1
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=info msg="loading plugin "io.containerd.grpc.v1.content"..." type=io.containerd.grpc.v1
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=info msg="loading plugin "io.containerd.grpc.v1.diff"..." type=io.containerd.grpc.v1
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=info msg="loading plugin "io.containerd.grpc.v1.events"..." type=io.containerd.grpc.v1
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=info msg="loading plugin "io.containerd.grpc.v1.healthcheck"..." type=io.containerd.grpc.v1
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=info msg="loading plugin "io.containerd.grpc.v1.images"..." type=io.containerd.grpc.v1
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=info msg="loading plugin "io.containerd.grpc.v1.leases"..." type=io.containerd.grpc.v1
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=info msg="loading plugin "io.containerd.grpc.v1.namespaces"..." type=io.containerd.grpc.v1
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=info msg="loading plugin "io.containerd.grpc.v1.snapshots"..." type=io.containerd.grpc.v1
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=info msg="loading plugin "io.containerd.grpc.v1.tasks"..." type=io.containerd.grpc.v1
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=info msg="loading plugin "io.containerd.grpc.v1.version"..." type=io.containerd.grpc.v1
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=info msg="loading plugin "io.containerd.grpc.v1.cri"..." type=io.containerd.grpc.v1
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=info msg="Start cri plugin with config {PluginConfig:{ContainerdConfig:{Snapshotter:overlayfs DefaultRuntime:{Type:io.containerd.runtime.v1.linux Engine: Root:} UntrustedWorkloadRuntime:{Type: Engine: Root:}} CniConfig:{NetworkPluginBinDir:/opt/cni/bin NetworkPluginConfDir:/etc/cni/net.d NetworkPluginConfTemplate:} Registry:{Mirrors:map[docker.io:{Endpoints:[https://registry-1.docker.io]}]} StreamServerAddress: StreamServerPort:10010 EnableSelinux:false SandboxImage:k8s.gcr.io/pause:3.1 StatsCollectPeriod:10 SystemdCgroup:false EnableTLSStreaming:false} ContainerdRootDir:/var/lib/containerd ContainerdEndpoint:/run/containerd/containerd.sock RootDir:/var/lib/containerd/io.containerd.grpc.v1.cri StateDir:/run/containerd/io.containerd.grpc.v1.cri}"
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=info msg="Connect containerd service"
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=info msg="Get image filesystem path "/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs""
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=error msg="Failed to load cni during init, please check CRI plugin status before setting up network for pods" error="cni config load failed: no network config found in /etc/cni/net.d: cni plugin not initialized: failed to load cni config"
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=info msg="loading plugin "io.containerd.grpc.v1.introspection"..." type=io.containerd.grpc.v1
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=info msg="Start subscribing containerd event"
Jul 18 02:46:25 cri-rhel-1.novalocal systemd[1]: containerd.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=info msg="Start recovering state"
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=info msg=serving... address="/run/containerd/containerd.sock"
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=info msg="containerd successfully booted in 0.004574s"
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=info msg="Start event monitor"
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=info msg="Start snapshots syncer"
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: time="2018-07-18T02:46:25-04:00" level=info msg="Start streaming server"
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: panic: runtime error: invalid memory address or nil pointer dereference
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x60 pc=0x557269b5b54e]
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: goroutine 75 [running]:
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server.(*snapshotsSyncer).sync(0xc42039be60, 0x55726a3bb0e0, 0xc420260410)
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/snapshots.go:79 +0x19e
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server.(*snapshotsSyncer).start.func1(0xc420260410, 0xc42039be60)
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/snapshots.go:62 +0x77
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: created by github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server.(*snapshotsSyncer).start
Jul 18 02:46:25 cri-rhel-1.novalocal containerd[12024]: /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/snapshots.go:57 +0x5d
Jul 18 02:46:25 cri-rhel-1.novalocal systemd[1]: Unit containerd.service entered failed state.
Jul 18 02:46:25 cri-rhel-1.novalocal systemd[1]: containerd.service failed.

My host info:

# cat /etc/*release
NAME="Red Hat Enterprise Linux Server"
VERSION="7.5 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.5"
PRETTY_NAME="Red Hat Enterprise Linux Server 7.5 Beta (Maipo)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.5:beta:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.5
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.5 Beta"
Red Hat Enterprise Linux Server release 7.5 Beta (Maipo)
Red Hat Enterprise Linux Server release 7.5 Beta (Maipo)

# cat /proc/version
Linux version 3.10.0-851.el7.x86_64 ([email protected]) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC) ) #1 SMP Mon Feb 12 07:53:52 EST 2018

Does the kernel version problem?When I install on another rhel 7.5 environment which kernel information:

cat /proc/version
Linux version 3.10.0-862.6.3.el7.x86_64 ([email protected]) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC) ) #1 SMP Fri Jun 15 17:57:37 EDT 2018

On this host containerd can work well.

# crictl --version
crictl version 1.0.0-beta.0
@Random-Liu
Copy link
Member

@hmtai

  1. The panic was fixed in containerd 1.1.1. Fix snapshotter nil panic. #826
  2. You should configure containerd/cri to use snapshot available on rehl, e.g. aufs? btrfs? (See snapshotter in https://github.com/containerd/cri/blob/master/docs/config.md)

@hmtai
Copy link
Author

hmtai commented Jul 19, 2018

Yes,when I use cri-containerd-1.1.1.linux-amd64.tar.gz,there is no panic but there is another error:

# crictl info
FATA[0000] getting status of runtime failed: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService

As my test on RHEL,I find that

@hmtai hmtai closed this as completed Jul 19, 2018
@hmtai
Copy link
Author

hmtai commented Jul 19, 2018 •
edited
Loading

Yes,when I use cri-containerd-1.1.1.linux-amd64.tar.gz,there is no panic but there is another error:

# crictl info
FATA[0000] getting status of runtime failed: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService

# containerd --version
containerd github.com/containerd/containerd v1.1.1 d64c661f1d51c48782c9cec8fda7604785f93587

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
kind/question
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Random-Liu @hmtai