[RFC] runc exec --cgroup

[kolyshkin]

Any container can have sub-cgroups

• created by the container itself, if access is provided (cgroupns is enabled and cgroup mount is not readonly -- see examples for cgroup v1 and v2);
• created by some other tools (higher-level orchestrators, systemd, etc);
• created by runc itself (if a future runtime-spec will add an ability to define sub-cgroups and their resources).

This is a proposal to add a feature to have runc exec executed in a sub-cgroup of a container, rather than its top-level cgroup as it happens now (except for cgroup v2, which has a fallback to join container init's cgroup).

For example:

runc exec -d --cgroup /foo/bar CID cmd args...

will run cmd args inside container CID, putting cmd in container's foo/bar cgroup, relative to container's top-level cgroup.

Obviously, the default value for --cgroup is /, which is how it's working now.

In a similar manner, runtime-spec's Process structure need to add a Cgroup field with the same meaning as runc exec's --cgroup. For container init it doesn't make sense to have Cgroup set to any value other than /. For other execs, it can be changed.

One other implementation detail is, I guess runc exec --cgroup should NOT create the sub-cgroup if it does not exist, but return an error.

[mrunalp]

The use case for this is dpdk applications that use a subset of the configured cpuset in the main container entirely and they can't tolerate k8s probes (runc exec) running on those cpus.

[AkihiroSuda]

How will this relate to subtree_controllers?

[kolyshkin]

How will this relate to subtree_controllers?

I think it should be orthogonal (except for the "runtime-spec will add an ability to define sub-cgroups" item which is actually not part of this proposal, and only given as an example).

The runc exec --cgroup merely uses the existing in-container cgroup, and errors out if it's not available.

[giuseppe]

how will this work with cgroup v2? The processes can be only in the leaf nodes, do we need to ensure the existing processes are moved to a sibling cgroup first?

[kolyshkin]

do we need to ensure the existing processes are moved to a sibling cgroup first?

I don't think it's a runc task (unless the sub-cgroups are created by runc itself, but that's not what this RFC proposes).

Currently if runc exec process fails to join the top-level container cgroup, it retries with the cgroup of the container init. I guess this fallback should be disabled when --cgroup is set explicitly.

[kolyshkin]

An initial implementation is available (#3059). Will work on more tests next week but it's good enough to take a look.

[giuseppe]

do we need to ensure the existing processes are moved to a sibling cgroup first?

I don't think it's a runc task (unless the sub-cgroups are created by runc itself, but that's not what this RFC proposes).

So does it expect the cgroupfs to be mounted writeable? Otherwise how would the container process live in a separate sub-cgroup?

[kolyshkin]

So does it expect the cgroupfs to be mounted writeable? Otherwise how would the container process live in a separate sub-cgroup?

Yes (in the future though we might implement sub-cgroup support in runtime spec, in which case runc will pre-create those sub-cgroups and writable cgroupfs won't be needed in case the cgroup tree is static).

[kolyshkin]

in the future though we might implement sub-cgroup support in runtime spec, in which case runc will pre-create those sub-cgroups and writable cgroupfs won't be needed in case the cgroup tree is static

Obviously the use case of that (with read-only cgroupfs) would be limited to putting container init in a non-top cgroup, and runc exec --cgroup (as the container won't be able to manage anything). Not sure it can be useful or not.