how do rancheros and k3os differ?

[rbucker]

k3os does not install with docker-machine and rancheros does. Also install.sh --takeover does not install over RancherOS... so, what do I need to install onto RancherOS to put it in parity with k3os?

(Consider this... CoreOS is strong because the host OD is nearly immutable. RanckerOS and k3os are more like standard server OS because they can install plenty of addons instead of pushing the addon to the container.)

[dweomer]

RancherOS has at it's core a Docker engine as the init system (system-docker) but also if you squint at is just so it is congruent with boot2docker. Think of it as a particular implementation of boot2docker because this is how it is utilized by docker-machine. Add-ons are largely compose-like compositions with write_files glue.

k3OS can be thought of, similarly, as boot2k3s. It is decidedly and explicitly NOT a containerized operating system (in contrast with with RancherOS or LinuxKit) but a container experience is readily available via k3s. Add-ons should be deployed as manifests which is to say that it's all just kubernetes. You still have the write_files glue combined with boot_cmd and run_cmd but these should be used only to get the "boot2k3s" experience running in a particular environment be it edge or cloud. Please be advised that k3OS should be thought of as immutable in that all changes that aren't explicitly on the persistent disk are ephemeral and will be lost with reboot.

[vincent99]

RancherOS is a minimal OS to get docker on a machine, and run all the system services as docker containers. It knows nothing about Kubernetes, but is a suitable target for RKE to create a kubernetes cluster out of.

k3os is a minimal OS to run k3s, that is to turn a machine into a Kubernetes cluster (or add it as another node to an existing one). Deploying k3os with docker-machine doesn't really make any sense.. there is no Docker in k3s or k3os. It is more like an alternative to using RKE to create a kubernetes cluster on a set of (often) bare-metal machines.

[dweomer]

As far as installing k3s on RancherOS, I believe that rancher/k3d might help but there is also this from @ibuildthecloud: k3s-io/k3s#73

[rbucker]

@dweomer @vincent99 these are great explanations and does sharpen my general understanding. What I was hoping for was a little ROS + TOOLS = K3os..... and I was trying to figure out what TOOLS I needed to install to get parity. (The larger problem I've posted in a number of places is the K3OS does not install with docker-machine and as far as I can tell will not take over a RancherOS installation.)

[vincent99]

Like I said docker-machine + k3s/k3os just doesn't really make a whole lot sense as a starting premise. Yes we still use it in Rancher, but it's largely abandonware as far as Docker is concerned and we are starting to pick up maintenance of it and/or working on alternatives for the actual VM provisioning.

The two things machine does are create a VM and install Docker on it, but you don't need or really want Docker for k3s (because it uses containerd). A good amount of effort goes into making k3s's footprint smaller, and RancherOS+system containers+docker are a decent chunk of unnecessary overhead on top of that (especially for a constrained system like a Raspberry Pi).

If you mainly like docker-machine for the actually creating the VM in a cloud provider part, you'd probably be better off with something like Ansible/Chef/Terraform. Or your-favorite-cloud's-CLI.. There's lots of tools out there that will make API calls to create a VM (or ISO loading, PXE booting, etc for baremetal). And k3os can be configured in various ways to auto-install itself from there.

[rbucker]

@vincent99 Thanks for that slap in the face. I think I read that someplace but maybe forgot or in denial... But since it seems that maybe I have your ear... Let me ask you a question. Have you every used Borland Turbo Pascal? Back in the early days when it was a simple console editor/compiler? Let me emphasize "simple".

In years past I built a complete vCloud orchestration system for my employer and it was brutal. The idea of hand creating multi-machine templates etc was a nightmare. But it worked and saved them millions. But there is something to be said for simple. All of those tools you mentioned are painful to configure, some are stupid expensive, and others are meant for a class of problem that few ever achieve. (see fossil vs git at fossil-scm.org) The gist of the article is that git was meant to be difficult and it was meant for huge numbers of developers and LOC. GIT is a google sized solution.

From a business development perspective the question is "does teraform help customers get to k8s, k3s, k3os or just add tons of complexity they likely did not get project funding for?

As it is docker-machine only installs a few number of OS and certainly a limited number of providers. (oh even though teraform has x-hundred providers it requires a ton of work to get there.)

PS: k3s-containerd + docker explanatino makes sense. How much work would it take to get docker-machine to install k3os? I think it's just sshd, and a shared uid/pwd.

[vincent99]

I'm not exactly a fan of say Terraform either and it is certainly more typing spread among multiple .tf files. It can do lots of complicated things, but the basics that are equivalent to what machine offers are not exactly rocket surgery. e.g. there's a DigitalOcean tutorial anybody could follow to get a few VMs up.

They also give you path for doing day-2 maintenance of the machines (if you care), or to do things d-m doesn't, or not do things it does do but you don't want. For example everyone hates d-m insisting on opening up security groups for SSH and the docker socket. Half the drivers have an option to use an existing SSH key but half don't. Etc.

You could certainly fork machine and add a new provisioner that creates an appropriate cloud-config, detects that the installation from ISO is k3os and then does totally different stuff.. the RancherOS one is actually somewhat like that, since it's not really installing Docker (which is already present).

But you're basically starting down the path of "Situation: there are 14 hard to use configuration management systems" (https://xkcd.com/927/). That's fine if you know that and just want to build exactly what you want for yourself. But if you want other people to use it you'll quickly discover those complexities exist for reasons and these things are a large part of the reason docker-machine never got used all that much beyond demos and is now collecting dust.

[rbucker]

See, now that's the "add generics to golang" argument. It is possible to be good enough, serve the most meaningful usecases and convince the yahoo who wants pink elephants that purple will get the job done it's just not pink.

[rbucker]

I guess this one is gone to ground. Thanks too all who contributed.