Skip to content

Commit

Permalink
install: add Minikube installation instructions
Browse files Browse the repository at this point in the history 
Add instructions of how to setup and install Kata under
Minikue.

Fixes: kata-containers#444

Signed-off-by: Graham Whaley <[email protected]>
  • Loading branch information
Graham Whaley committed Jun 11, 2019
1 parent df57d05 commit a5eb3aa
Showing 1 changed file with 240 additions and 0 deletions.
240 changes: 240 additions & 0 deletions install/minikube-installation-guide.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,240 @@
# Installing Kata Containers in [Minikube](https://kubernetes.io/docs/setup/minikube/)

* [Installing Kata Containers in Minikube](#installing-kata-containers-in-minikube)
* [Introduction](#introduction)
* [Prerequisites](#prerequisites)
* [Setting up Minikube](#setting-up-minikube)
* [Checking for nested virtualization](#checking-for-nested-virtualization)
* [Check Minikube is running](#check-minikube-is-running)
* [Installing Kata Containers](#installing-kata-containers)
* [Enabling Kata Containers](#enabling-kata-containers)
* [Register the runtime](#register-the-runtime)
* [Testing Kata Containers](#testing-kata-containers)
* [Wrapping up](#wrapping-up)

## Introduction

[Minikube](https://kubernetes.io/docs/setup/minikube/) is an easy way to try out a Kubernetes (k8s)
cluster locally. It utilises running a single node Kubernetes stack in a local VM.

[Kata Containers](https://github.com/kata-containers) can be installed into a Minikube cluster using
[kata-deploy](https://github.com/kata-containers/packaging/tree/master/kata-deploy).

This document details the pre-requisites, installation steps, and how to check
the installation has been successful.

## Prerequisites

This installation guide has only been verified under a Minikube Linux installation, using the
[kvm2](https://github.com/kubernetes/minikube/blob/master/docs/drivers.md#kvm2-driver) driver.

This installation guide may not work for MacOS installations of Minikube, due to the lack of
nested virtualization support on that platform.

This installation guide has not been tested on a Windows installation.

See the [Minikube installation guide](https://kubernetes.io/docs/tasks/tools/install-minikube/)
for details on installation, whilst paying attention to the following [Setup](#setting-up-minikube)
section of this document. Also pay attention to the following
[Nested Virtualization](#checking-for-nested-virtualization) section of this document, as
Kata Containers will not function under Minikube if this is not enabled on your system.

## Setting up Minikube

To enable Kata Containers under Minikube, you need to add a few configuration options to the
default Minikube setup. This is easily accomplished, as Minikube supports them on the setup commandline.

Here are the features, and why you need them:

| what | why |
| ---- | --- |
| `--vm-driver kvm2` | The host VM driver |
| `--memory 6144` | Allocate sufficient memory, as Kata Containers default to 1 or 2Gb |
| `--network-plugin=cni` | As recommended for [minikube CRI-o](https://kubernetes.io/docs/setup/minikube/#cri-o) |
| `--enable-default-cni` | As recommended for [minikube CRI-o](https://kubernetes.io/docs/setup/minikube/#cri-o) |
| `--container-runtime=cri-o` | Using CRI-O for Kata |
| `--bootstrapper=kubeadm` | As recommended for [minikube CRI-o](https://kubernetes.io/docs/setup/minikube/#cri-o) |

> **Note:** This example deploys Kata Containers using CRI-O. The same procedure should also work
> for a `containerd` based Kubernetes installation, but is not covered in this document.
> **Note:** Prior to Minikube/Kubernetes v1.14, the beta `RuntimeClass` feature also needed enabling with
> the following. Recent Minikube/Kubernetes installations should no longer require this setting.
>
> | what | why |
> | ---- | --- |
> | `--feature-gates=RuntimeClass=true` | Kata needs to use the RuntimeClass Kubernetes feature |
The full command is therefore:

```sh
$ minikube start --vm-driver kvm2 --memory 6144 --network-plugin=cni --enable-default-cni --container-runtime=cri-o --bootstrapper=kubeadm
```

> **Note:** For Kata Containers >v1.6.2, the now default `tcfilter` networking of Kata Containers
> did not work for Minikube versions <v1.1.1. Please ensure you use Minikube version >=v1.1.1.
## Checking for nested virtualization

For Kata Containers to work under a Minikube VM, your host system must support
nested virtualization. If you are using a Linux system utilising Intel VT-x
and the `kvm_intel` driver, you can perform the following check:

```sh
$ cat /sys/module/kvm_intel/parameters/nested
Y
```

If your system does not report `Y` from the `nested` parameter, then details on how
to enable nested virtualisation can be found on the
[KVM Nested Guests page](https://www.linux-kvm.org/page/Nested_Guests)

## Check Minikube is running

Before you install Kata Containers, perform a quick check that your Minikube is
operating. On your guest:


```sh
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
minikube Ready master 78m v1.13.4
```

Check you have virtualization enabled inside your Minikube:

```sh
# No color option, as minikube is running busybox
$ minikube ssh "egrep 'vmx|svm' /proc/cpuinfo"

# if you get a long line of output here, you have it enabled!
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology cpuid pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single pti retpoline rsb_ctxsw tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 xsaves arat umip

# A failed result means it is not enabled:
ssh: Process exited with status 1
```

## Installing Kata Containers

You can now install the Kata Containers runtime components. You will need a local copy of some Kata
Containers components to help with this, and then use the host `kubectl` (that Minikube has already
configured for you) to deploy them:

```sh
$ git clone https://github.com/kata-containers/packaging.git
$ cd packaging/kata-deploy
$ kubectl apply -f kata-rbac.yaml
$ kubectl apply -f kata-deploy.yaml
```

This installs the Kata Containers components into `/opt/kata` inside the Minikube node. It can take
a few minutes for the operation to complete. You can check the installation has worked by checking
the status of the `kata-deploy` pod. This can be accomplished by running the following:

```sh
$ podname=$(kubectl -n kube-system get pods -o=name | fgrep kata-deploy | sed 's?pod/??')
# If 'sleep infinity' is running, the pod has completed its work.
$ kubectl -n kube-system exec ${podname} -- ps -ef | fgrep infinity
root 49 1 0 10:55 ? 00:00:00 sleep infinity
```

> *NOTE:* This check only works for single node clusters, which is the default for Minikube.
> For multi-node clusters, the check would need to be adapted to check `kata-deploy` had
> completed on all nodes.
## Enabling Kata Containers

> **Note:** This step is only required for Minikube/Kubernetes versions <= 1.13. Since version
> v1.14, the `RuntimeClass` is enabled by default. Performing this step on Kubernetes > v1.14 is
> however benign.
Now the Kata Containers components are installed in the Minikube node, you need to configure
Kubernetes `RuntimeClass` to know when to use Kata Containers to run a pod.

```sh
$ curl https://raw.githubusercontent.com/kubernetes/node-api/master/manifests/runtimeclass_crd.yaml > runtimeclass_crd.yaml
$ kubectl apply -f runtimeclass_crd.yaml
```

### Register the runtime

Now register the `kata qemu` runtime with that class:

```sh
$ cd packaging/kata-deploy/k8s-1.14
$ kubectl apply -f kata-qemu-runtimeClass.yaml
runtimeclass.node.k8s.io/kata created
runtimeclass.node.k8s.io/kata-qemu created
```

Kata Containers should now be installed and enabled in the Minikube cluster.

## Testing Kata Containers

Launch a container that is defined to run on Kata Containers. The enabling is configured by
the following lines in the yaml file:

```yaml
spec:
runtimeClassName: kata-qemu
```
Perform the following action to launch a Kata Containers based pod:
```sh
$ cd packaging/kata-deploy/examples
$ kubectl apply -f test-deploy-kata-qemu.yaml
```

An Apache PHP pod should have been deployed with the Kata Containers runtime. Wait a few moments
to check it is running:

```sh
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
php-apache-kata-qemu-bc4c55994-p6f6k 1/1 Running 0 1m
```

There are a couple of ways to verify it is running with Kata Containers. Nominally, it
should be hard to tell - the idea of Kata Containers is that your pod will run inside a VM,
but look and feel just like it would as a normal software pod. In
theory, you should not be able to tell your pod is running as a Kata Containers container.
Careful examination can verify your pod is in fact a Kata Containers pod.

First, look on the node for a `qemu` running, and which version of the kernel the
node is running:

```sh
# You should see a VERY long line of qemu here - showing that you are indeed running
# a qemu VM on the Minikube node - that is the VM that contains your pod.
$ minikube ssh -- pgrep -a qemu

# For use in a moment, see what kernel is running on the node itself.
$ minikube ssh -- uname -a
Linux minikube 4.15.0 #1 SMP Wed Mar 6 23:18:58 UTC 2019 x86_64 GNU/Linux
```

You should have seen a QEMU process running, indicating that your pod was indeed
running inside a Kata Containers VM.

Another way to verify Kata Containers is running is to look in the container itself and check
which kernel is running there. For a normal software container you will be running
the same kernel as the node. For a Kata Container you will be running a Kata Containers kernel
inside the Kata Containers VM.

```sh
$ podname=$(kubectl get pods -o=name | fgrep php-apache-kata-qemu | sed 's?pod/??')
$ kubectl exec ${podname} -- uname -a
Linux php-apache-kata-qemu-7d4647498f-5cl9q 4.19.28 #1 SMP Wed Jun 5 14:32:34 PDT 2019 x86_64 GNU/Linux
```

There you should see, the node is running kernel 4.15, but the container running under Kata Containers
is running (in this instance) a 4.19 kernel.

## Wrapping up

This guide should have shown a relatively easy way to get a Minikube up with Kata Containers installed.
Be aware, this is only a small single node Kubernetes cluster running under a nested virtualization setup.
It will have limitations, but, as a first introduction to Kata Containers, and how to install it under Kubernetes,
it should suffice for initial learning and experimentation.

0 comments on commit a5eb3aa

Please sign in to comment.