k8s v1.24.0-alpha.1: failed to run Kubelet: using dockershim is not supported

[tstromberg]

What Happened?

minikube at HEAD (90300a4) defaults to using dockershim, which no longer ships with Kubernetes v1.24.x. One can work around this by overriding the default container runtime, for instance, by passing in --container-runtime=containerd.

However, this isn't an ideal user experience.

More specifically, minikube start --kubernetes-version=v1.24.0-alpha.1 yields the following error:

💢  initialization failed, will try again: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.24.0-alpha.1:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml  --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,Mem": Process exited with status 1

minikube logs shows that the kubelet is in a boot-loop because it's expecting dockershim, which no longer ships with v1.24.0:

Dec 09 21:42:10 minikube kubelet[4450]: I1209 21:42:10.101298    4450 server.go:445] "Kubelet version" kubeletVersion="v1.24.0-alpha.1"
Dec 09 21:42:10 minikube kubelet[4450]: I1209 21:42:10.102905    4450 server.go:871] "Client rotation is on, will bootstrap in background"
Dec 09 21:42:10 minikube kubelet[4450]: I1209 21:42:10.107272    4450 certificate_store.go:130] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-current.pem".
Dec 09 21:42:10 minikube kubelet[4450]: I1209 21:42:10.110271    4450 dynamic_cafile_content.go:156] "Starting controller" name="client-ca-bundle::/var/lib/minikube/certs/ca.crt"
Dec 09 21:42:10 minikube kubelet[4450]: I1209 21:42:10.111315    4450 server.go:692] "--cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /"
Dec 09 21:42:10 minikube kubelet[4450]: I1209 21:42:10.111483    4450 container_manager_linux.go:281] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Dec 09 21:42:10 minikube kubelet[4450]: I1209 21:42:10.111562    4450 container_manager_linux.go:286] "Creating Container Manager object based on Node Config" nodeConfig={RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:docker CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:systemd KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: ReservedSystemCPUs: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[]} QOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalCPUManagerPolicyOptions:map[] ExperimentalTopologyManagerScope:container ExperimentalCPUManagerReconcilePeriod:10s ExperimentalMemoryManagerPolicy:None ExperimentalMemoryManagerReservedMemory:[] ExperimentalPodPidsLimit:-1 EnforceCPULimits:true CPUCFSQuotaPeriod:100ms ExperimentalTopologyManagerPolicy:none}
Dec 09 21:42:10 minikube kubelet[4450]: I1209 21:42:10.111602    4450 topology_manager.go:133] "Creating topology manager with policy per scope" topologyPolicyName="none" topologyScopeName="container"
Dec 09 21:42:10 minikube kubelet[4450]: I1209 21:42:10.111613    4450 container_manager_linux.go:321] "Creating device plugin manager" devicePluginEnabled=true
Dec 09 21:42:10 minikube kubelet[4450]: I1209 21:42:10.111638    4450 state_mem.go:36] "Initialized new in-memory state store"
Dec 09 21:42:10 minikube kubelet[4450]: E1209 21:42:10.111682    4450 server.go:301] "Failed to run kubelet" err="failed to run Kubelet: using dockershim is not supported, please consider using a full-fledged CRI implementation"
Dec 09 21:42:10 minikube systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Dec 09 21:42:10 minikube systemd[1]: kubelet.service: Failed with result 'exit-code'.

Full log file: log.txt

Recommendations

On at least Kubernetes v1.24.0+, minikube should select an alternative default CRI implementation.

In the meantime, passing --container-runtime=containerd allows minikube to run Kubernetes v1.24.0-alpha.1.

Related: kubernetes/kubernetes#106917

[afbjorklund]

See https://github.com/Mirantis/cri-dockerd:

• Use cri-dockerd instead of dockershim where needed #12102
• Add cri-dockerd (and crictl) to installation #12103

docker@minikube:~$ more /etc/crictl.yaml 
runtime-endpoint: unix:///var/run/cri-docker.sock
image-endpoint: unix:///var/run/cri-docker.sock
docker@minikube:~$ sudo systemctl list-units cri-docker.*
  UNIT               LOAD   ACTIVE SUB     DESCRIPTION                                          

  cri-docker.service loaded active running CRI Interface for Docker Application Container Engine
  cri-docker.socket  loaded active running CRI Docker Socket for the API                        

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

2 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.

[afbjorklund]

This is duplicate of:

• Start cri-docker service automatically, if using docker and kubernetes version >= 1.24 #12099

[afbjorklund]

However, this isn't an ideal user experience.

It also breaks minikube docker-env support.

[tstromberg]

Closing as dupe of #12099