Add support for K3d + Sysbox

[rodnymolina]

The goal here is to allow K3d tool to be able to create K8s clusters running over Sysbox containers. Sysbox would offer the following benefits over K3d's default runtime (runc):

• Security: Sysbox would allow secure containers to be utilized as K8s-nodes -- currently K3d can only run over 'privileged' containers.
• Simplicity: K3d's Dockerfile images could be drastically simplified as Sysbox absorbs most of the complexity required to allow K8s execution.
• Flexibility: Sysbox imposes no restrictions on the docker images to utilize, so users would have more flexibility to define their own (customized) K8s-node images, including inner images corresponding to K8s components and/or applications.

[ctalledo]

Of the benefits listed, I think the security benefit would certainly be realizable and is the most compelling.

Regarding the simplicity & flexibility benefits, these may not be realizable given that k3d must work with the OCI runc which requires more complex container images for the K8s nodes. It's not clear to me that k3d can be easily changed to use the simpler images which would work with sysbox but not with the OCI runc. A similar problem occurs with integrating K8s.io KinD + sysbox. This problem does not occur with kindbox since it's designed to work with sysbox, thus allowing users to use any container image for the k8s nodes.

[iwilltry42]

Now I just stumbled upon sysbox and then this issue and it sounds pretty interesting!
@rodnymolina & @ctalledo , what changes would be required on k3d side?
I assume it'd need

• flag to disable the privileged mode
• flag to use sysbox
• simplified K3s image (that can be chosen using k3d's --image flag)
  Probably those options could be bundled into a single feature flag 🤔

[rodnymolina]

Hi @iwilltry42, glad that you found us. And btw, thanks for your quick fix for this and this, will test it later.

For the record: I stumbled into that problem while trying to run K3d inside a Sysbox-powered container, which is also part of the scope of this K3d+Sysbox integration effort.

Now, in regards to this issue's primary goal (run K3d through Sysbox runtime), I think you pretty much described all the tasks that would be needed. And yes, I agree that from a UX perspective, the best would be to bundle these features in a single K3d flag.

Please let us know if have any other question. For obvious reasons, we are interested in having this job done asap, so let us know if you need any help from us.

[iwilltry42]

Hi @rodnymolina , sorry, this got buried in my mailbox.. 🙄
With regards to the simplified K3s image: do you have an image already that would work with Sysbox?

[ctalledo]

Hi @iwilltry42, thanks for following up.

With regards to the simplified K3s image: do you have an image already that would work with Sysbox?

In theory any image that carries a distro supported by K3s should work. That is, you should be able to do docker run --runtime=sysbox-runc <some-image> to create a (rootless) Sysbox container, and from inside that image run curl -sfL https://get.k3s.io | sh - and K3s should come up (as if it where running in a VM or bare-metal).

For example, this works:

# Launch a Sysbox container 
$ docker run --runtime=sysbox-runc -d --rm --name=k3s-master nestybox/ubuntu-focal-systemd-docker

# Exec into it and start K3s as usual
$ docker exec -it k3s-master /bin/bash

root@8d11757f7be1:/# cd
root@8d11757f7be1:~# curl -sfL https://get.k3s.io | sh -
[INFO]  Finding release for channel stable
[INFO]  Using v1.22.5+k3s1 as release
[INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.22.5+k3s1/sha256sum-amd64.txt
[INFO]  Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.22.5+k3s1/k3s
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Skipping installation of SELinux RPM
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Skipping /usr/local/bin/ctr symlink to k3s, command exists in PATH at /usr/bin/ctr
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO]  systemd: Starting k3s

root@8d11757f7be1:~# kubectl get all
NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.43.0.1    <none>        443/TCP   73s

# Verify container is rootless 
root@8d11757f7be1:~# cat /proc/self/uid_map 
         0     165536      65536                   # <<< root user in container maps to unprivileged user in host

The Dockerfile for nestybox/ubuntu-focal-systemd-docker is here.

Hope this helps, let me know of any other questions please.