cgroupv2: don't enable threaded mode by default

[lifubang]

After this commit: 60c647e
Runc enable threaded mode in cgroup v2 by default.
If the cgroupPath is set to a absolute path like /docker/********, the memory subsystem can't be used by this mode.
So, I think we should use domain mode by default.

Signed-off-by: lifubang [email protected]

[lifubang]

@AkihiroSuda PTAL

[cyphar]

I'm not sure I understand why we're putting cgroupv2 into threaded mode -- @AkihiroSuda is this to work around some odd permission issue (and why does it fix an EPERM)? The description in 60c647e and #2340 is pretty sparse.

[AkihiroSuda]

cgType := filepath.Join(current, "cgroup.type")
cgTypeB, _ := ioutil.ReadFile(cgType)
if strings.TrimSpace(string(cgTypeB)) == "domain invalid" {
  _ = ioutil.WriteFile(cgType, []byte("threaded"), 0644)
}

[AkihiroSuda]

@cyphar
Writing threaded to cgroup.type is required when cgroup.type becomes domain invalid, mostly in nested environment

$ sudo podman run -it --rm --privileged --runtime=crun alpine
/ # cd /sys/fs/cgroup/
/sys/fs/cgroup # cat cgroup.controllers 
cpuset cpu io memory pids
/sys/fs/cgroup # cat cgroup.subtree_control 
/sys/fs/cgroup # echo +cpu > cgroup.subtree_control
/sys/fs/cgroup # mkdir foo
/sys/fs/cgroup # cat foo/cgroup.type
domain invalid

[cyphar]

Okay, but "domain invalid" means that the cgroup is in an invalid state (meaning that one of the cgroup rules has been violated -- most likely the no-internal-processes rule). Putting a cgroup into threaded mode doesn't fix that -- it switches it into an alternative mode which only allows controllers which are thread-aware to be enabled (such as cpu). This is why we can't enable the memory controller -- it isn't thread-aware.

IMHO, a more complete solution would be to figure out how to deal with the parent cgroup having child processes (which is a bit dodgy if we're going to move other programs on the system between cgroups) or to simply give an error if we hit domain invalid and explain what the issue is.

[lifubang]

or to simply give an error if we hit domain invalid and explain what the issue is.

Personally, I prefer this one. @AkihiroSuda WDYT? Because in man7 cgroups:

2. "Internal" processes are not permitted.  With the exception of the
          root cgroup, processes may reside only in leaf nodes (cgroups that
          do not themselves contain child cgroups).  The details are
          somewhat more subtle than this, and are described below.

Cgroups v2 "no internal processes" rule
       Cgroups v2 enforces a so-called "no internal processes" rule.
       Roughly speaking, this rule means that, with the exception of the
       root cgroup, processes may reside only in leaf nodes (cgroups that do
       not themselves contain child cgroups).  This avoids the need to
       decide how to partition resources between processes which are members
       of cgroup A and processes in child cgroups of A.

       For instance, if cgroup /cg1/cg2 exists, then a process may reside in
       /cg1/cg2, but not in /cg1.  This is to avoid an ambiguity in cgroups
       v1 with respect to the delegation of resources between processes in
       /cg1 and its child cgroups.  The recommended approach in cgroups v2
       is to create a subdirectory called leaf for any nonleaf cgroup which
       should contain processes, but no child cgroups.  Thus, processes
       which previously would have gone into /cg1 would now go into
       /cg1/leaf.  This has the advantage of making explicit the relation‐
       ship between processes in /cg1/leaf and /cg1's other children.

       The "no internal processes" rule is in fact more subtle than stated
       above.  More precisely, the rule is that a (nonroot) cgroup can't
       both (1) have member processes, and (2) distribute resources into
       child cgroups—that is, have a nonempty cgroup.subtree_control file.
       Thus, it is possible for a cgroup to have both member processes and
       child cgroups, but before controllers can be enabled for that cgroup,
       the member processes must be moved out of the cgroup (e.g., perhaps
       into the child cgroups).

       With the Linux 4.14 addition of "thread mode" (described below), the
       "no internal processes" rule has been relaxed in some cases.

[AkihiroSuda]

SGTM, but when no domain controller is enabled, we can write "threaded" without retuning error

[AkihiroSuda]

IIUC we don't need to check parent, we just need to check whether the current config contains domain controller

[lifubang]

runc/tests/rootless.sh

Line 103 in 1d14356

echo threaded > "$CGROUP_MOUNT/$CGROUP_PATH/cgroup.type"

I think rootless doesn't need threaded mode in default?
I don't know whether my opinion is right or not. @AkihiroSuda

[AkihiroSuda]

Let's check invalid cgroup.type and set threaded conditionally

[AkihiroSuda]

You can drop L106-L110 now

[kolyshkin]

@lifubang ^^^

[lifubang]

This has been removed before LGTM.
I think we should keep these comments to let other people know why we need to write threaded to cgroups.type.

[cyphar]

Is the echo threaded > still needed? Seems a bit odd to run the entire test suite under threaded.

[AkihiroSuda]

LGTM

[kolyshkin]

a few minor fixes, otherwise good

[AkihiroSuda]

LGTM

[AkihiroSuda]

@kolyshkin LGTY?

[cyphar]

LGTM.

issues fixed

[giuseppe]

@AkihiroSuda @kolyshkin PTAL

[kolyshkin]

LGTM