Should BurmillaOS support kubernetes?

[tomaswarynyca]

I open this issues so that they give arguments that why we should support kubernetes in the project.
Specify concrete cases of use so we can make the best decision and if you give affirmative it helps us to make the documentation in a future.

[zimme]

I actually created a custom system service for k3s which I got up and running which basically gave me a lightweight k8s distro running on top of the user-docker instance. Worked fine expect for when restarting the service; when k3s server was running with the --docker flag it was bringing down the user-docker instance with it when shutting down during a restart of the service which stopped all of my other runnint containers in user-docker.

As I wasn't able to find the problem where k3s server --docker was using the docker socket to close the docker daemon too I abandoned the experiment.

If this could be solved then having a k3s service could be doable.

[olljanat]

Adding k3s with --docker switch at least does not make any sense as support for it will be removed on near future. (Look: #22 (comment) )

Packaging k3s + containerd to own separated system service most probably would works but it leaves open the question that what is the point? Why not using just k3OS then?

Also the reality is that currently I'm doing most of the BurmillaOS maintenance my self on my spare time which puts some limitations to that what is really doable. So minimum requirement that we are able to support Kubernetes is find someone who will do most of the maintenance related to that one but then again it is quite unclear for me that why anyone would do that as they can just participate k3OS maintenance?

[Raboo]

What does it mean by "support kubernetes"?
I'm running Kubernetes on RancherOS and are planning to jump over to BurmillaOS in the future.

[olljanat]

What does it mean by "support kubernetes"?

@Raboo if we decide that we do not support Kubernetes it means that all bug reports which cannot be repeated without Kubernetes will be closed with "wontfix" label. It also means that we do not need to care about fact that it might be impossible run Kubernetes top of RancherOS/BurmillaOS after their CRI migration is ready and Docker support is removed https://www.theregister.com/AMP/2020/12/03/docker_deprecated_in_kubernetes_this/

I'm running Kubernetes on RancherOS and are planning to jump over to BurmillaOS in the future.

As short term solution that most probably works just fine but I still highly recommend for all Kubernetes users to investigate other distros (e.g. k3OS, Fedora CoreOS, etc) which focuses to support Kubernetes also on long term.

PS. Company where I'm working on is nowadays also piloting RKE2 as long term replacement for Docker Swarm and will most probably use it with "RKE2 OS" if Rancher eventually releases it (currently there is only rumor that it might happen).

[wonleing]

Also the reality is that currently I'm doing most of the BurmillaOS maintenance my self on my spare time which puts some limitations to that what is really doable. So minimum requirement that we are able to support Kubernetes is find someone who will do most of the maintenance related to that one but then again it is quite unclear for me that why anyone would do that as they can just participate k3OS maintenance?

We are very happy to help on maintenaning this project and do contribution, since it could be part of our business.
The problem is we don't have enough doc/info how to replace/upgrade kernel/rootfs/initrd/service in this project.
Actually, I have tried some days on build a new kernel with source code from kernel.org, but the whole system can not boot up. Could you please share us the process of upgrading kernel and rootfs etc.?
My mail is [email protected], please drop me a mail, and let's make this project more active!

[olljanat]

@wonleing plz join to our Discord #10 (comment) where trying to have some kind of project leading discussion.

[olljanat]

@wonleing btw also plz share more details about your use case(s) on #6

[olljanat]

For everyone who is planning to run Kubernetes on top of BurmillaOS on longterm. Please check my new architecture proposal on #88

[wonleing]

To me, running Kubernetes/K3S on top of BurmillaOS is a MUST have feature for Bare Metal solution.
So I would take effort to make this happen
But I don't mean K8s/K3s should be built-in. I prefer it works as an extention.

[olljanat]

@wonleing btw. I just remembered that feat/k3s-support branch contains draft version of K3s support which I once started to implement but don't remember which state it.

Idea on that one was test if it is possible to support enabling K3s as parallel with user docker. Service configuration exist on https://github.com/burmilla/os-services/tree/feat/k3s

[pwFoo]

BurmillaOS should native use docker. That's why I use BurmillaOS instead of k3os.
k3s as a Burmilla system-docker service would be nice addition to user-docker.

[prologic]

I agree with @pwFoo here. This is why I've used RancherOS all along, but going to upgrade to BurmillaOS due to recent changes in Alpine that cause havoc with Make and Docker as described here

If first-class for k8s or k3s is added to BurmillaOS, I'd be fine with that as long as it didn't break any existing use-cases or functionality and is/was completely optional. I'd also not want to see the ISO size increase just to support this, as that (for me) would incur additional resource costs I don't need or want.

[olljanat]

FYI for those who are running Kubernetes on RancherOS/BurmillaOS: Recently released Kubernetes 1.23 does not support Docker anymore.

In theory it might be possible to continue to use Docker still as runtime with https://github.com/Mirantis/cri-dockerd but I highly recommend to consider containerd instead of.

[prologic]

Oh well 😂 No loss for me, let's close this issue 😅

[olljanat]

FYI, new interesting option for BurmillaOS users who want utilize K8s tooling without maintaining actusl K8s https://www.k2d.io (created by Portainer team).