diff --git a/install/minikube-installation-guide.md b/install/minikube-installation-guide.md new file mode 100644 index 00000000..ca44f401 --- /dev/null +++ b/install/minikube-installation-guide.md @@ -0,0 +1,231 @@ +# Installing Kata Containers in Minikube + +* [Installing Kata Containers in Minikube](#installing-kata-containers-in-minikube) + * [Introduction](#introduction) + * [Prerequisites](#prerequisites) + * [Setting up Minikube](#setting-up-minikube) + * [Checking for nested virtualization](#checking-for-nested-virtualization) + * [Check Minikube is running](#check-minikube-is-running) + * [Installing Kata Containers](#installing-kata-containers) + * [Enabling Kata Containers](#enabling-kata-containers) + * [Register the runtime](#register-the-runtime) + * [Testing Kata Containers](#testing-kata-containers) + * [Wrapping up](#wrapping-up) + +## Introduction + +[Minikube](https://kubernetes.io/docs/setup/minikube/) is an easy way to try out a Kubernetes (k8s) +cluster locally. It creates a single node Kubernetes stack in a local VM. + +[Kata Containers](https://github.com/kata-containers) can be installed into a Minikube cluster using +[`kata-deploy`](https://github.com/kata-containers/packaging/tree/master/kata-deploy). + +This document details the pre-requisites, installation steps, and how to check +the installation has been successful. + +## Prerequisites + +This installation guide has only been verified under a Minikube Linux installation, using the +[`kvm2`](https://github.com/kubernetes/minikube/blob/master/docs/drivers.md#kvm2-driver) driver. + +> **Notes:** +> - This installation guide may not work for macOS installations of Minikube, due to the lack of +nested virtualization support on that platform. +> - This installation guide has not been tested on a Windows installation. + +Before commencing installation, it is strongly recommended you read the +[Minikube installation guide](https://kubernetes.io/docs/tasks/tools/install-minikube/). + +## Checking for nested virtualization + +For Kata Containers to work under a Minikube VM, your host system must support +nested virtualization. If you are using a Linux system utilizing Intel VT-x +and the `kvm_intel` driver, you can perform the following check: + +```sh +$ cat /sys/module/kvm_intel/parameters/nested +``` + +If your system does not report `Y` from the `nested` parameter, then details on how +to enable nested virtualization can be found on the +[KVM Nested Guests page](https://www.linux-kvm.org/page/Nested_Guests) + +Alternatively, and for other architectures, the Kata Containers built in +[`kata-check`](https://github.com/kata-containers/runtime#hardware-requirements) +command can be used *inside Minikube* once Kata has been installed, to check for compatibility. + +## Setting up Minikube + +To enable Kata Containers under Minikube, you need to add a few configuration options to the +default Minikube setup. You can easily accomplish this as Minikube supports them on the setup commandline. + +Here are the features, and why you need them: + +| what | why | +| ---- | --- | +| `--bootstrapper=kubeadm` | As recommended for [minikube CRI-o](https://kubernetes.io/docs/setup/minikube/#cri-o) | +| `--container-runtime=cri-o` | Using CRI-O for Kata | +| `--enable-default-cni` | As recommended for [minikube CRI-o](https://kubernetes.io/docs/setup/minikube/#cri-o) | +| `--memory 6144` | Allocate sufficient memory, as Kata Containers default to 1 or 2Gb | +| `--network-plugin=cni` | As recommended for [minikube CRI-o](https://kubernetes.io/docs/setup/minikube/#cri-o) | +| `--vm-driver kvm2` | The host VM driver | + +> **Notes:** +> - Adjust the `--memory 6144` line to suit your environment and requirements. Kata Containers default to +> requesting 2048MB per container. We recommended you supply more than that to the Minikube node. +> - This example deploys Kata Containers using CRI-O. The same procedure should also work +> for a `containerd` based Kubernetes installation, but we do not cover that in this document. +> +> - Prior to Minikube/Kubernetes v1.14, the beta `RuntimeClass` feature also needed enabling with +> the following. +> +> | what | why | +> | ---- | --- | +> | `--feature-gates=RuntimeClass=true` | Kata needs to use the `RuntimeClass` Kubernetes feature | + +The full command is therefore: + +```sh +$ minikube start --vm-driver kvm2 --memory 6144 --network-plugin=cni --enable-default-cni --container-runtime=cri-o --bootstrapper=kubeadm +``` + +> **Note:** For Kata Containers later than v1.6.1, the now default `tcfilter` networking of Kata Containers +> does not work for Minikube versions less than v1.1.1. Please ensure you use Minikube version v1.1.1 +> or above. + +## Check Minikube is running + +Before you install Kata Containers, check that your Minikube is operating. On your guest: + + +```sh +$ kubectl get nodes +``` + +You should see your `master` node listed as being `Ready`. + +Check you have virtualization enabled inside your Minikube. The following should return +a number larger than `0` if you have either of the `vmx` or `svm` nested virtualization features +available: + +```sh +$ minikube ssh "egrep -c 'vmx|svm' /proc/cpuinfo" +``` + +## Installing Kata Containers + +You can now install the Kata Containers runtime components. You will need a local copy of some Kata +Containers components to help with this, and then use `kubectl` on the host (that Minikube has already +configured for you) to deploy them: + +```sh +$ git clone https://github.com/kata-containers/packaging.git +$ cd packaging/kata-deploy +$ kubectl apply -f kata-rbac.yaml +$ kubectl apply -f kata-deploy.yaml +``` + +This installs the Kata Containers components into `/opt/kata` inside the Minikube node. It can take +a few minutes for the operation to complete. You can check the installation has worked by checking +the status of the `kata-deploy` pod, which will be executing +[this script](https://github.com/kata-containers/packaging/blob/master/kata-deploy/scripts/kata-deploy.sh), +and will be executing a `sleep infinity` once it has successfully completed its work. +You can accomplish this by running the following: + +```sh +$ podname=$(kubectl -n kube-system get pods -o=name | fgrep kata-deploy | sed 's?pod/??') +$ kubectl -n kube-system exec ${podname} -- ps -ef | fgrep infinity +``` + +> *NOTE:* This check only works for single node clusters, which is the default for Minikube. +> For multi-node clusters, the check would need to be adapted to check `kata-deploy` had +> completed on all nodes. + +## Enabling Kata Containers + +> **Note:** Only Minikube/Kubernetes versions <= 1.13 require this step. Since version +> v1.14, the `RuntimeClass` is enabled by default. Performing this step on Kubernetes > v1.14 is +> however benign. + +Now you have installed the Kata Containers components in the Minikube node. Next, you need to configure +Kubernetes `RuntimeClass` to know when to use Kata Containers to run a pod. + +```sh +$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/node-api/master/manifests/runtimeclass_crd.yaml > runtimeclass_crd.yaml +``` + +### Register the runtime + +Now register the `kata qemu` runtime with that class. This should result in no errors: + +```sh +$ cd packaging/kata-deploy/k8s-1.14 +$ kubectl apply -f kata-qemu-runtimeClass.yaml +``` + +The Kata Containers installation process should be complete and enabled in the Minikube cluster. + +## Testing Kata Containers + +Launch a container that has been defined to run on Kata Containers. The enabling is configured by +the following lines in the YAML file. See the Kubernetes +[Runtime Class Documentation](https://kubernetes.io/docs/concepts/containers/runtime-class/#usage) +for more details. + +```yaml + spec: + runtimeClassName: kata-qemu +``` + +Perform the following action to launch a Kata Containers based Apache PHP pod: + +```sh +$ cd packaging/kata-deploy/examples +$ kubectl apply -f test-deploy-kata-qemu.yaml +``` + +This may take a few moments if the container image needs to be pulled down into the cluster. +Check progress using: + +```sh +$ kubectl rollout status deployment php-apache-kata-qemu +``` + +There are a couple of ways to verify it is running with Kata Containers. +In theory, you should not be able to tell your pod is running as a Kata Containers container. +Careful examination can verify your pod is in fact a Kata Containers pod. + +First, look on the node for a `qemu` running. You should see a QEMU command line output here, +indicating that your pod is running inside a Kata Containers VM: + +```sh +$ minikube ssh -- pgrep -a qemu +``` + +Another way to verify Kata Containers is running is to look in the container itself and check +which kernel is running there. For a normal software container you will be running +the same kernel as the node. For a Kata Container you will be running a Kata Containers kernel +inside the Kata Containers VM. + +First, examine which kernel is running inside the Minikube node itself: + +```sh +$ minikube ssh -- uname -a +``` + +And then compare that against the kernel that is running inside the container: + +```sh +$ podname=$(kubectl get pods -o=name | fgrep php-apache-kata-qemu | sed 's?pod/??') +$ kubectl exec ${podname} -- uname -a +``` + +You should see the node and pod are running different kernel versions. + +## Wrapping up + +This guide has shown an easy way to setup Minikube with Kata Containers. +Be aware, this is only a small single node Kubernetes cluster running under a nested virtualization setup. +As such, it has limitations, but as a first introduction to Kata Containers, and how to install it under Kubernetes, +it should suffice for initial learning and experimentation. +