Error while running a docker container: shim error: open pid: no such file or directory

[padhia]

Submission type

• [ X] Bug report
• Feature Request

Package name & version you are having issues with or have a request for

docker-1.12.3-1, containerd-0.2.4-1, runc-0.1.1-2

Used repositories

core, main, apps,

Expected behaviour

Docker should be able to start containers

Actual behaviour

container doesn't start. systemctl status docker shows following messages
Nov 23 10:53:56 td-linux dockerd[16045]: time="2016-11-23T10:53:56.577585843-05:00" level=error msg="containerd: start container" error="shim error: open pid: no such file or directory" id=ff1a3d15d5958f7
Nov 23 10:53:56 td-linux dockerd[16045]: time="2016-11-23T10:53:56.581128100-05:00" level=error msg="Create container failed with error: shim error: open pid: no such file or directory"
Nov 23 10:53:56 td-linux dockerd[16045]: time="2016-11-23T10:53:56.629877616-05:00" level=warning msg="Couldn't run auplink before unmount /var/lib/docker/aufs/mnt/d0fb6043c7994cf5ecfbbc80e651171c6795d043
Nov 23 10:53:56 td-linux dockerd[16045]: time="2016-11-23T10:53:56.643734123-05:00" level=error msg="Handler for POST /v1.24/containers/ff1a3d15d5958f70e10e8abcc9c0b92dad6e74322b187350245b1be898221c54/sta
Nov 23 11:03:00 td-linux dockerd[16045]: time="2016-11-23T11:03:00.778953069-05:00" level=warning msg="Couldn't run auplink before unmount /var/lib/docker/aufs/mnt/1d8f9208b581a5de7448c70181391e7dbed4bd4d
Nov 23 11:03:00 td-linux dockerd[16045]: time="2016-11-23T11:03:00.805314336-05:00" level=warning msg="Couldn't run auplink before unmount /var/lib/docker/aufs/mnt/1d8f9208b581a5de7448c70181391e7dbed4bd4d
Nov 23 11:03:00 td-linux dockerd[16045]: time="2016-11-23T11:03:00.86870113-05:00" level=error msg="containerd: start container" error="shim error: open pid: no such file or directory" id=24c4fda96ac61f2c
Nov 23 11:03:00 td-linux dockerd[16045]: time="2016-11-23T11:03:00.871685648-05:00" level=error msg="Create container failed with error: shim error: open pid: no such file or directory"
Nov 23 11:03:00 td-linux dockerd[16045]: time="2016-11-23T11:03:00.914920932-05:00" level=warning msg="Couldn't run auplink before unmount /var/lib/docker/aufs/mnt/1d8f9208b581a5de7448c70181391e7dbed4bd4d
Nov 23 11:03:00 td-linux dockerd[16045]: time="2016-11-23T11:03:00.935161658-05:00" level=error msg="Handler for POST /v1.24/containers/24c4fda96ac61f2cba8c5540900a603028cccf55c6c8e15787ad131c13076c7e/sta

Steps to reproduce the problem

docker pull alpine
docker run --rm -ti alpine bash

System info link created by Octopi > Tools > SysInfo

https://gist.github.com/3ccf098138d586c840e000f72e3e5faa

Additional Information

I believe this error was previously resolved by building runc using a specific commit
moby/moby#23958

[demmm]

No, this looks like an issue with cgroup settings in systemd. I have been following bug reports for that in both systemd & docker. (there are workarounds listed upstream).
Looks like systemd added a patch to make cgroup changes optional.
opencontainers/runc#1175
moby/moby#28109
systemd/systemd#4628

Either way, this is not KaOS but upstream. Not sure what the course needs to be, adjust systemd or wait/see when docker gets ready.

[demmm]

Opted for reverting the cgroup change in systemd. Patched systemd is up in [build], please note if that fixes this for you.

sudo pacman -U http://kaosx.tk/repo/build/systemd-1%3A232-2-x86_64.pkg.tar.xz

[padhia]

It's not reporting the original error, but docker client doesn't seem to be able to communicate with the daemon even though docker service is up and running.

~> docker run --rm -ti alpine bash
docker: Cannot connect to the Docker daemon. Is the docker daemon running on this host?.
See 'docker run --help'.
~> sudo systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: enabled)
   Active: active (running) since Wed 2016-11-23 12:11:14 EST; 6min ago
     Docs: https://docs.docker.com
 Main PID: 960 (dockerd)
    Tasks: 27 (limit: 4915)
   Memory: 53.6M
      CPU: 261ms
   CGroup: /system.slice/docker.service
           ├─960 /usr/bin/dockerd -H fd://
           └─995 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --shim docker-containerd-shim --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libconta

Nov 23 12:11:14 td-linux dockerd[960]: time="2016-11-23T12:11:14.206613988-05:00" level=info msg="[graphdriver] using prior storage driver \"aufs\""
Nov 23 12:11:14 td-linux dockerd[960]: time="2016-11-23T12:11:14.244068498-05:00" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Nov 23 12:11:14 td-linux dockerd[960]: time="2016-11-23T12:11:14.244541278-05:00" level=info msg="Loading containers: start."
Nov 23 12:11:14 td-linux dockerd[960]: .time="2016-11-23T12:11:14.261135165-05:00" level=info msg="Firewalld running: false"
Nov 23 12:11:14 td-linux dockerd[960]: time="2016-11-23T12:11:14.352151149-05:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to 
Nov 23 12:11:14 td-linux dockerd[960]: time="2016-11-23T12:11:14.383350692-05:00" level=info msg="Loading containers: done."
Nov 23 12:11:14 td-linux dockerd[960]: time="2016-11-23T12:11:14.383717699-05:00" level=info msg="Daemon has completed initialization"
Nov 23 12:11:14 td-linux dockerd[960]: time="2016-11-23T12:11:14.383740784-05:00" level=info msg="Docker daemon" commit=6b644ec graphdriver=aufs version=1.12.3
Nov 23 12:11:14 td-linux dockerd[960]: time="2016-11-23T12:11:14.387841116-05:00" level=info msg="API listen on /var/run/docker.sock"
Nov 23 12:11:14 td-linux systemd[1]: Started Docker Application Container Engine.

[demmm]

Sorry, this does not seem to be a docker issue but your used image. Internet shows many places alpine does not use bash as default.
So, looks like the original bug report is solved, docker is running.

[padhia]

You are right about missing bash. I realized that alpine just uses busybox, so sh works instead of bash.

Anyways, I was able to get docker working after building it from git. So the modified systemd from the build repo and docker from git is a working combination for me. Still not sure why docker from the KaOS repo won't work. If no one else has problems with docker from the repos, maybe something's wrong with my installation.

[demmm]

Nope, I can reproduce now. Docker runs and can do some commands, but to make it actually work it fails with pid errors.
If docker git fixes it then will use that.
Did you get a new runc version too?

[padhia]

Yes, I got git versions of all 3 components: runc, containerd and docker