[BUG] Failed to start ContainerManager cannot enter cgroupv2 "/sys/fs/cgroup/kubepods" with domain controllers -- it is in an invalid state

[boniek83]

What did you do

$ k3d cluster create test
INFO[0000] Prep: Network
INFO[0000] Created network 'k3d-test' (9980e94b5ab55e9c933046774cbdc9ccccb86880cb4c86a5580a1fe40a96f17b)
INFO[0000] Created volume 'k3d-test-images'
INFO[0001] Creating node 'k3d-test-server-0'
INFO[0001] Creating LoadBalancer 'k3d-test-serverlb'
INFO[0001] Starting cluster 'test'
INFO[0001] Starting servers...
INFO[0001] Starting Node 'k3d-test-server-0'
INFO[0068] Starting agents...
INFO[0068] Starting helpers...
INFO[0068] Starting Node 'k3d-test-serverlb'
INFO[0068] (Optional) Trying to get IP of the docker host and inject it into the cluster as 'host.k3d.internal' for easy access
WARN[0071] Failed to patch CoreDNS ConfigMap to include entry '172.31.0.1 host.k3d.internal': Exec process in node 'k3d-test-server-0' failed with exit code '137'
INFO[0071] Successfully added host record to /etc/hosts in 2/2 nodes
INFO[0071] Cluster 'test' created successfully!
INFO[0071] --kubeconfig-update-default=false --> sets --kubeconfig-switch-context=false
INFO[0071] You can now use it like this:
kubectl config use-context k3d-test
kubectl cluster-info

/sys/fs/cgroup/kubepods doesn't exist at any point in time

What did you expect to happen

That it works if it says Cluster 'test' created successfully! ;)

Screenshots or terminal output

https://gist.github.com/boniek83/f1f0d3b4741d06b828f357ea70e0fd82

Which OS & Architecture

Fedora 34 amd64

Which version of k3d

$ k3d version
k3d version v4.4.2
k3s version v1.20.6-k3s1 (default)

Which version of docker

$ sudo docker version
Client:
Version: 20.10.5
API version: 1.41
Go version: go1.16
Git commit: 55c4c88
Built: Sun Mar 14 22:41:08 2021
OS/Arch: linux/amd64
Context: default
Experimental: true

Server:
Engine:
Version: 20.10.5
API version: 1.41 (minimum version 1.12)
Go version: go1.16
Git commit: 363e9a8
Built: 2021-03-14 22:40:26.546445533+00:00
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.5.0~rc.1
GitCommit:
runc:
Version: 1.0.0-rc93
GitCommit: 48389904331fdebba5e2db73147c067560a51d80
docker-init:
Version: 0.19.0
GitCommit:

$ sudo docker info
Client:
Context: default
Debug Mode: false

Server:
Containers: 2
Running: 1
Paused: 0
Stopped: 1
Images: 13
Server Version: 20.10.5
Storage Driver: overlay2
Backing Filesystem: btrfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: journald
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: /usr/libexec/docker/docker-init
containerd version:
runc version: 48389904331fdebba5e2db73147c067560a51d80
init version:
Security Options:
seccomp
Profile: default
selinux
cgroupns
Kernel Version: 5.11.15-300.fc34.x86_64
Operating System: Fedora 34 (Workstation Edition)
OSType: linux
Architecture: x86_64
CPUs: 12
Total Memory: 31.36GiB
Name: zen
ID: E4PM:7SEB:M23S:OMJG:K7ZL:IYEN:EXKX:UVOH:4UM2:SE4X:LANA:OOSH
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: true

WARNING: Support for cgroup v2 is experimental

[iwilltry42]

Hi @boniek83 , thanks for opening this issue!
Funny enough I just released https://github.com/rancher/k3d/releases/tag/v4.4.3 today that ships cgroupv2 support (set the environment variable K3D_FIX_CGROUPV2=1).
Can you please give this a try?
References:

• https://github.com/rancher/k3d/releases/tag/v4.4.3
• fix/workaround: add workaround for cgroupv2 until fixed in k3s #579
• [BUG] Cluster fails to start on cgroup v2 #493

[boniek83]

It seems to work. Thanks!