kubelet failed to create "/system" cgroup

[yujuhong]

On a debian-based containervm node, kubelet no long creates the "/system" cgroup and moves processes into it.

From kubelet.log:

W1020 16:16:51.129436    3552 container_manager_linux.go:491] [ContainerManager] Failed to ensure state of "/system": [failed to list PIDs for root: open cgroup.procs: no such file or directory, ran out of attempts to create system containers "/system"]

/cc @kubernetes/sig-node

[yujuhong]

By the way, we have poor to no test coverage on features like this.

[derekwaynecarr]

isnt that running systemd? you shouldn't do any of this anymore on systemd nodes.

[yujuhong]

@derekwaynecarr there are two bugs (I didn't include enough information in the original issue):

1. From what I observed, kubelet still tries to create this on the new GCI image (with non-init systemd), and fails repeatedly. Kubelet should not try to create this at all.
2. On the old debian-based container vm image (i.e., no systemd) kubelet should create this cgroup, but it fails to do so.

[yujuhong]

@dchen1107 should we fix this for 1.5?

[vishh]

FYI: #35319 should fix the GCI
aspects of this issue.

On Thu, Oct 27, 2016 at 11:35 AM, Yu-Ju Hong [email protected]
wrote:

@dchen1107 https://github.com/dchen1107 should we fix this for 1.5?

—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub
#35214 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGvIKLEzCKZmZGcEycX2nIJfmtDDzEnMks5q4O8BgaJpZM4KcX6U
.

[yujuhong]

This regression is caused by a change in libcontainerd: opencontainers/runc#1013
The old code calls getCgroupPath, but the new code relies on Manager.Paths being populated beforehand.

I looked around a little bit and we have a CgroupManager in kubelet that wraps around the library, but the Pids() method it provides seems to be recursive (which is probably not what we need). I am not familiar with the the code and not sure whether we should reuse any of the existing code. @derekwaynecarr and @vishh will have better suggestions.

[vishh]

I can take a look at this today.

On Wed, Nov 9, 2016 at 12:25 PM, Yu-Ju Hong [email protected]
wrote:

This regression is caused by a change in libcontainerd:
opencontainers/runc#1013
opencontainers/runc#1013
The old code calls getCgroupPath, but the new code relies on Manager.Paths
being populated beforehand.

I looked around a little bit and we have a CgroupManager
https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/cm/cgroup_manager_linux.go
in kubelet that wraps around the library, but the Pids()

kubernetes/pkg/kubelet/cm/cgroup_manager_linux.go

Line 397 in 42289c2

func (m *cgroupManagerImpl) Pids(name CgroupName) []int {

method it provides seems to be recursive (which is probably not what we
need). I am not familiar with the the code and not sure whether we should
reuse any of the existing code. @derekwaynecarr
https://github.com/derekwaynecarr and @vishh https://github.com/vishh
will have better suggestions.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#35214 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGvIKKjTpR-d5dVCyfC3X0oq4l1TrcQoks5q8iwwgaJpZM4KcX6U
.

[timstclair]

Spoke with Vish, I'll take this. I'm going to try and get a patch by tomorrow that we can cherry-pick into 1.4.6 (cc @jessfraz )