Skip to content
This repository has been archived by the owner on Jan 4, 2022. It is now read-only.

docker/runc doesn't work on Fedora 26 #45

Closed
robertgzr opened this issue Jun 22, 2017 · 5 comments
Closed

docker/runc doesn't work on Fedora 26 #45

robertgzr opened this issue Jun 22, 2017 · 5 comments

Comments

@robertgzr
Copy link
Contributor

robertgzr commented Jun 22, 2017 •
edited
Loading

Full kubelet error:

kubeadm-nspawn-0 kubelet[82]: E0622 17:23:33.029071      82 pod_workers.go:182] Error syncing pod 7075157cfd4524dbe0951e00a8e3129e ("etcd-kubeadm-nspawn-0_kube-system(7075157cfd4524dbe0951e00a8e3129e)"), skipping: failed to "CreatePodSandbox" for "etcd-kubeadm-nspawn-0_kube-system(7075157cfd4524dbe0951e00a8e3129e)" with CreatePodSandboxError: "CreatePodSandbox for pod \"etcd-kubeadm-nspawn-0_kube-system(7075157cfd4524dbe0951e00a8e3129e)\" failed: rpc error: code = 2 desc = failed to start sandbox container for pod \"etcd-kubeadm-nspawn-0\": Error response from daemon: {\"message\":\"invalid header field value \\\"oci runtime error: container_linux.go:247: starting container process caused \\\\\\\"process_linux.go:359: container init caused \\\\\\\\\\\\\\\"rootfs_linux.go:53: mounting \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"cgroup\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\" to rootfs \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"/var/lib/docker/overlay/c211c41b8e46e1cdd7def0298d675db90eb563a6fac0d17b7d32db71264ea979/merged\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\" at \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"/sys/fs/cgroup\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\" caused \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"no subsystem for mount\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"\\\\\\\\\\\\\\\"\\\\\\\"\\\\n\\\"\"}"

Reproduce: docker run --rm -it busybox inside coreos container spawned by kubeadm-nspawn on Fedora 26

docker: Error response from daemon: invalid header field value "oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:359: container init caused \\\"rootfs_linux.go:53: mounting \\\\\\\"cgroup\\\\\\\" to rootfs \\\\\\\"/var/lib/docker/overlay/550589b3e61fe73d1e5fdba641f562796e056c3ba92e877a04562dc34fe8a237/merged\\\\\\\" at \\\\\\\"/sys/fs/cgroup\\\\\\\" caused \\\\\\\"no subsystem for mount\\\\\\\"\\\"\"\n".

inside the coreos container:

  • docker v1.12.6
  • systemd v233

Possible ways to fix:

  • change cgroup-driver of kubelet/docker to cgroupfs and upgrade runc to at least 1.0rc3 (default docker on coreos uses runc 1.0rc2)
  • turn off off unified cgroup hierarchy with systemd.legacy_systemd_cgroup_controller=1
@robertgzr
Copy link
Contributor Author

/cc @alban

@alban
Copy link
Member

alban commented Jun 23, 2017

Could we specify systemd.legacy_systemd_cgroup_controller=1 in the Vagrantfile?

@robertgzr
Copy link
Contributor Author

robertgzr commented Jun 28, 2017 •
edited
Loading

The docker command above works on the fedora host with docker v1.13.1 and systemd v233

Latest docker should have the fix to the runc issue: opencontainers/runc#1175 (comment)

@robertgzr
Copy link
Contributor Author

robertgzr commented Jun 28, 2017 •
edited
Loading

Can be fixed by switching to CoreOS Alpha v1451.2.0 -> see

This was referenced Jul 6, 2017
Closed
Closed
robertgzr added a commit that referenced this issue Jul 10, 2017
@robertgzr
Update documentation
1ef7aa0 
Split up the README to only include general information and on how to run the
project.

Needs #30 #45 #49

Closes #51
@robertgzr robertgzr mentioned this issue Jul 10, 2017
Merged
@robertgzr
Copy link
Contributor Author

Fixed via #32

dongsupark pushed a commit that referenced this issue Jul 13, 2017
@dongsupark
etc: change cgroup driver to cgroupfs
3cb2b69 
With runc 1.0.0-rc2 on Container Linux 1465, kube-spawn init hangs
forever with message like: "Created API client, waiting for the
control plane to become ready".
That's because docker daemon cannot execute runc, which returns error
like "no subsystem for mount". See also:
opencontainers/runc#1175 (comment)

This issue was apparently resolved in runc 1.0.0-rc3, so in theory
runc 1.0.0-rc3 should work fine with Docker 17.05. Unfortunately on
Container Linux, it's not trivial to replace only the runc binary with
a custom one, because Container Linux makes use of torcx to provide
docker as well as runc: /run/torcx/unpack is sealed, read-only mounted.
It's simply not doable to change those binaries altogether at run-time.

As workaround, we should change cgroupdriver for docker and kubelet
from systemd to cgroupfs. Then init process will succeed without hanging
forever.

See also #45
dongsupark pushed a commit that referenced this issue Jul 13, 2017
@dongsupark
etc: change cgroup driver to cgroupfs
879c11f 
With runc 1.0.0-rc2 on Container Linux 1465, kube-spawn init hangs
forever with message like: "Created API client, waiting for the
control plane to become ready".
That's because docker daemon cannot execute runc, which returns error
like "no subsystem for mount". See also:
opencontainers/runc#1175 (comment)

This issue was apparently resolved in runc 1.0.0-rc3, so in theory
runc 1.0.0-rc3 should work fine with Docker 17.05. Unfortunately on
Container Linux, it's not trivial to replace only the runc binary with
a custom one, because Container Linux makes use of torcx to provide
docker as well as runc: /run/torcx/unpack is sealed, read-only mounted.

As workaround, we should change cgroupdriver for docker and kubelet
from systemd to cgroupfs. Then init process will succeed without hanging
forever.

See also #45
@dongsupark dongsupark mentioned this issue Jul 13, 2017
Merged
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@alban @robertgzr