failed to run Kubelet: invalid configuration: cgroup-root ["kubelet"] doesn't exist

[powerman]

What happened: kind create cluster fails on "Starting control-plane"

What you expected to happen: It should not fail.

How to reproduce it (as minimally and precisely as possible):
I belive this is related to a way how cgroups are mounted on my system.
I'm using Gentoo with runit, and mount cgroups this way:

mount -n -t tmpfs -o "noexec,nosuid,nodev,mode=755,size=10m" cgroup_root /sys/fs/cgroup
mkdir -p /sys/fs/cgroup/unified
mount -t cgroup2 cgroup2_root -o "noexec,nosuid,nodev,nsdelegate" /sys/fs/cgroup/unified
while read -r name _ _ enabled _; do
	case "${enabled}" in
	1)
		mkdir /sys/fs/cgroup/"${name}"
		mount -n -t cgroup -o "noexec,nosuid,nodev,${name}" "${name}" /sys/fs/cgroup/"${name}"
		;;
	esac
done </proc/cgroups

which results in:

# mount | grep cgroup
cgroup_root on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,relatime,size=10240k,mode=755)
cgroup2_root on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)
cpuset on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cpu on /sys/fs/cgroup/cpu type cgroup (rw,nosuid,nodev,noexec,relatime,cpu)
cpuacct on /sys/fs/cgroup/cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct)
blkio on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
memory on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
devices on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
freezer on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
net_cls on /sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls)
perf_event on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
net_prio on /sys/fs/cgroup/net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_prio)
hugetlb on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
pids on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
rdma on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma)
cgroup on /sys/fs/cgroup/elogind type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib64/elogind/elogind-cgroups-agent,name=elogind)

Anything else we need to know?:
Command output:

 ✗ Starting control-plane 🕹️ 
ERROR: failed to create cluster: failed to init node with kubeadm: command "docker exec --privileged kind-control-plane kubeadm init --skip-phases=preflight --config=/kind/kubeadm.conf --skip-token-print --v=6" failed with error: exit status 1
Command Output: I0827 16:19:19.329570     225 initconfiguration.go:246] loading configuration from "/kind/kubeadm.conf"
[config] WARNING: Ignored YAML document with GroupVersionKind kubeadm.k8s.io/v1beta2, Kind=JoinConfiguration
[init] Using Kubernetes version: v1.21.1
...
I0827 16:19:20.915623     225 waitcontrolplane.go:87] [wait-control-plane] Waiting for the API server to be healthy
I0827 16:19:20.916065     225 loader.go:372] Config loaded from file:  /etc/kubernetes/admin.conf
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
I0827 16:19:20.916793     225 round_trippers.go:454] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I0827 16:19:21.417654     225 round_trippers.go:454] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
...
I0827 16:20:05.917946     225 round_trippers.go:454] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.
I0827 16:20:06.417944     225 round_trippers.go:454] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
...
I0827 16:20:15.917285     225 round_trippers.go:454] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.
I0827 16:20:16.417355     225 round_trippers.go:454] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
...
I0827 16:21:15.917867     225 round_trippers.go:454] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.

	Unfortunately, an error has occurred:
		timed out waiting for the condition

kind-control-plane/kubelet.log:

-- Journal begins at Fri 2021-08-27 16:13:17 UTC, ends at Fri 2021-08-27 16:15:25 UTC. --
Aug 27 16:13:18 kind-control-plane systemd[1]: Condition check resulted in kubelet: The Kubernetes Node Agent being skipped.
Aug 27 16:13:20 kind-control-plane systemd[1]: Starting kubelet: The Kubernetes Node Agent...
Aug 27 16:13:20 kind-control-plane systemd[1]: Started kubelet: The Kubernetes Node Agent.
Aug 27 16:13:20 kind-control-plane kubelet[275]: Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 27 16:13:20 kind-control-plane kubelet[275]: Flag --provider-id has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 27 16:13:20 kind-control-plane kubelet[275]: Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 27 16:13:20 kind-control-plane kubelet[275]: Flag --cgroup-root has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 27 16:13:20 kind-control-plane kubelet[275]: I0827 16:13:20.133746     275 server.go:197] "Warning: For remote container runtime, --pod-infra-container-image is ignored in kubelet, which should be set in that remote runtime instead"
Aug 27 16:13:20 kind-control-plane kubelet[275]: Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 27 16:13:20 kind-control-plane kubelet[275]: Flag --provider-id has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 27 16:13:20 kind-control-plane kubelet[275]: Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 27 16:13:20 kind-control-plane kubelet[275]: Flag --cgroup-root has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 27 16:13:20 kind-control-plane kubelet[275]: I0827 16:13:20.466197     275 server.go:440] "Kubelet version" kubeletVersion="v1.21.1"
Aug 27 16:13:20 kind-control-plane kubelet[275]: I0827 16:13:20.466362     275 server.go:851] "Client rotation is on, will bootstrap in background"
Aug 27 16:13:20 kind-control-plane kubelet[275]: I0827 16:13:20.468911     275 dynamic_cafile_content.go:167] Starting client-ca-bundle::/etc/kubernetes/pki/ca.crt
Aug 27 16:13:20 kind-control-plane kubelet[275]: I0827 16:13:20.468925     275 container_manager_linux.go:995] "CPUAccounting not enabled for process" pid=275
Aug 27 16:13:20 kind-control-plane kubelet[275]: I0827 16:13:20.468935     275 container_manager_linux.go:998] "MemoryAccounting not enabled for process" pid=275
Aug 27 16:13:20 kind-control-plane kubelet[275]: E0827 16:13:20.473809     275 certificate_manager.go:437] Failed while requesting a signed certificate from the master: cannot create certificate signing request: Post "https://kind-control-plane:6443/apis/certificates.k8s.io/v1/certificatesigningrequests": dial tcp 172.19.12.4:6443: connect: connection refused
Aug 27 16:13:22 kind-control-plane kubelet[275]: E0827 16:13:22.562477     275 certificate_manager.go:437] Failed while requesting a signed certificate from the master: cannot create certificate signing request: Post "https://kind-control-plane:6443/apis/certificates.k8s.io/v1/certificatesigningrequests": dial tcp 172.19.12.4:6443: connect: connection refused
Aug 27 16:13:25 kind-control-plane kubelet[275]: E0827 16:13:25.490069     275 server.go:292] "Failed to run kubelet" err="failed to run Kubelet: invalid configuration: cgroup-root [\"kubelet\"] doesn't exist"
Aug 27 16:13:25 kind-control-plane systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Aug 27 16:13:25 kind-control-plane systemd[1]: kubelet.service: Failed with result 'exit-code'.

Environment:

• kind version: (use kind version): kind v0.11.1 go1.17 linux/amd64
• Kubernetes version: (use kubectl version): Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.1", GitCommit:"632ed300f2c34f6d6d15ca4cef3d3c7073412212", GitTreeState:"clean", BuildDate:"2021-08-19T15:45:37Z", GoVersion:"go1.16.7", Compiler:"gc", Platform:"linux/amd64"}
• Docker version: (use docker info):

Client:
 Context:    default
 Debug Mode: false

Server:
 Containers: 4
  Running: 4
  Paused: 0
  Stopped: 0
 Images: 50
 Server Version: 20.10.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: active
  NodeID: zpsuptnxy3ft8y62wezq8vd2c
  Is Manager: true
  ClusterID: 4i0gm0730fwsd1340x3mbprcm
  Managers: 1
  Nodes: 1
  Default Address Pool: 10.0.0.0/8  
  SubnetSize: 24
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 3
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: 192.168.2.1
  Manager Addresses:
   192.168.2.1:2377
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7eba5930496d9bbe375fdf71603e610ad737d2b2
 runc version: 84113eef6fc27af1b01b3181f31bbaf708715301
 init version: de40ad007797e0dcd8b7126f27bb87401d224240
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.10.52-gentoo
 Operating System: Gentoo/Linux
 OSType: linux
 Architecture: x86_64
 CPUs: 24
 Total Memory: 31.36GiB
 Name: home
 ID: KYEC:QF2P:LCGZ:G3PY:QWZE:VDLU:FWRS:G4RB:XHNQ:SDJS:5XS6:7V3R
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 66
  Goroutines: 178
  System Time: 2021-08-27T19:27:29.721950652+03:00
  EventsListeners: 0
 Username: powerman
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Default Address Pools:
   Base: 172.19.0.0/16, Size: 24
   Base: 172.20.0.0/14, Size: 24
   Base: 172.24.0.0/13, Size: 24

• OS (e.g. from /etc/os-release): Gentoo

[powerman]

FWIW: minikube v1.22.0 works just fine using the docker driver.

[BenTheElder]

https://github.com/kubernetes-sigs/kind/issues?q=is%3Aissue+is%3Aopen+invalid+configuration%3A+cgroup-root

#2323

/triage duplicate

#2390 will land soon as a fix for #2323