services.kubernetes does not allow other runtimes than docker

[pinpox]

NOTE: This is not strictly a bug, didn't know which issue template would be the most fitting.

Describe the bug
Kubernetes can use different container runtimes, one of them being docker [1]. The support for docker has been deprecated in kubernetes [2] so while it technically still can be used, it will be a common setup to use containerd instead.
The services.kubernetes service has no option to set a runtime, but instead sets up docker when kubelet.enable is true.

This makes the service basically unusable if you want to use something other than docker. I would like to have support for containerd aswell. How much work would have to be done to support it? Is just adding an additional config option to select a backend enough? Are there other incompatibilities that have to be kept in mind?

I'm considering submitting a PR to add support for other runtimes, but I'm not sure how significant this change will have to be. If I can get some hints on the steps needed to add this to nixpkgs specifically, I'd be glad to help out (and scratch my own itch 😉 )

To Reproduce
Steps to reproduce the behavior:

1. services.kubernetes.enable = true
2. Try to use containerd instead of docker

Expected behavior
There should be an option to select the runtime that kubernetes will use to start containers

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context

[1] https://kubernetes.io/docs/setup/production-environment/container-runtimes/
[2] https://kubernetes.io/blog/2020/12/02/dont-panic-kubernetes-and-docker/
[3]

nixpkgs/nixos/modules/services/cluster/kubernetes/default.nix

Line 225 in c7d0dbe

virtualisation.docker = {

Notify maintainers
@johanot @saschagrunert @rnhmjoj @worldofpeace @infinisil @offlinehacker

Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

• system: "x86_64-linux"
• host os: Linux 5.4.99, NixOS, 21.05.20210217.6b1057b (Okapi)
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.4pre20201205_a5d85d0
• channels(root): ""
• channels(pinpox): "home-manager"
• nixpkgs: /nix/store/srvplqq673sqd9vyfhyc5w1p88y1gfm4-source

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute: services.kubernetes
# a list of nixos modules affected by the problem
module: kubernetes

[johanot]

Thanks for the issue! You're absolutely right. For the record, I'm working on the same right now, since we need it anyway for the kubernetes upgrade: #109275. In fact, Kubernetes is currently broken in nixos-unstable, since Docker 20.X removed dockershim. So yeah, tldr; we need this, and we'll also likely need to ship kubernetes.enable without docker entirely as a default, because of the official docker deprecation, as you've mentioned and linked in above.

[johanot]

Something along the lines of this, maybe: johanot@046185c .. However, instead of using docker embedded containerd as default, I'd prefer plain containerd. Of course, still with option for opt-out of the defaults.

[uhthomas]

Hi, just commenting here as this is relevant to a project I'm working on.

The official CRI of Kubernetes is CRI-O. There's already a nixpkg for CRI-O so it would be great if there was native support, or better yet having it as the default.

[saschagrunert]

The official CRI of Kubernetes is CRI-O.

To be fair, there is also containerd as alternative to mention.

There's already a nixpkg for CRI-O so it would be great if there was native support, or better yet having it as the default.

Yeah, I was working on it but then got side tracked. 🙃

[johanot]

Isn't a NixOS-module for CRI-O the only thing missing then? Since the latest changes to the Kubernetes module, it is in fact possible to configure custom runtimes.

I believe this issue should be closed or at least the title should change, in case we wanna repurpose the issue for implementing CRI-o support. :)

[johanot]

I believe this issue was fixed with the introduction of services.kubernetes.kubelet.containerRuntime
And
services.kubernetes.kubelet.containerRuntimeEndpoint ..

These options were introduced after this issue was opened.

I vote for closing this one.

Feel free to open new issues if you experience problems with any CRI-compliant runtime.

[saschagrunert]

Ah yeah let's close this one. :)

[uhthomas]

Great, thank you both. Glad to see progress being made on NixOS Kubernetes.

Are there any examples of how to use a CRI other than containerd? Is it possible for Kubernetes to not pull in containerd if it's unused?