diff --git a/docs/kubernetes/pouch_with_kubernetes_deploying.md b/docs/kubernetes/pouch_with_kubernetes_deploying.md index 0db010ad7..3a1d9ee8a 100644 --- a/docs/kubernetes/pouch_with_kubernetes_deploying.md +++ b/docs/kubernetes/pouch_with_kubernetes_deploying.md @@ -1,87 +1,148 @@ # Deploy Kubernetes With PouchContainer -Updated: 2018.6.1 - -- [PouchContainer deploying](#pouch-with-kubernetes-deploying) - - [Overview](#overview) - - [Restriction](#restriction) - - [Install and Configure](#install-and-configure) - - [Install PouchContainer](#install-pouch) - - [Setup Repo](#setup-repo) - - [Install Kubernetes Components](#install-kubernetes-components) - - [Install CNI](#install-cni) - - [Setting up the master node](#setting-up-the-master-node) - - [Setting up the minion nodes](#setting-up-the-minion-nodes) - - [Run and Verify](#run-and-verify) - - [Troubleshooting](#troubleshooting) +Updated: July 26th, 2018 + +- [Overview](#overview) +- [Prerequisite](#Prerequisite) + - [Software Version Requirement](#software-version-requirement) + - [Node Resource Requirement](#node-resource-requirement) +- [Install and Configure PouchContainer](#install-and-configure-PouchContainer) + - [Install PouchContainer](#install-pouchcontainer) + - [Configure PouchContainer](#configure-pouchContainer) + - [Restart PouchContainer](#restart-pouchcontainer) + - [Verify PouchContainer Correctness](#verify-pouchcontainer-correctness) +- [Install and Configure Kubernetes](#install-and-configure-kubernetes) + - [Download Kubernetes Packages](#Download-Kubernetes-Packages) + - [Configure Kubernetes Components](#Configure-Kubernetes-Components) + - [Setup Kubernetes Cluster](#setup-kubernetes-cluster) + - [Verify Kubernetes Correctness](#verify-Kubernetes-Correctness) +- [Setup CNI network](#Setup-CNI-network) + - [flannel](#flannel) + - [Calico](#Calico) ## Overview -This document shows how to easily install a Kubernetes cluster with PouchContainer as the container runtime.For Chinese customer, it's might be faster and easier to download Kubernetes components and various kinds of container images from Aliyun [Kubernetes + PouchContainer + Aliyun](pouch_with_kubernetes_deploying_aliyun.md). +This document shows how to easily install a Kubernetes cluster with PouchContainer as the container runtime. For Chinese customers, it might be faster and easier to download Kubernetes components and various kinds of container images from Aliyun [Kubernetes + PouchContainer + Aliyun](pouch_with_kubernetes_deploying_aliyun.md). ![pouch_with_kubernetes](../static_files/pouch_with_kubernetes.png) -## Restriction +This document is telling users how to deploy a Kubernetes cluster with specific steps. If you are looking for an all-in-one script to install all components of Kubernetes on a single node, refer to [allinone.sh](../../hack/kubernetes/allinone.sh). -Kubernetes: Version 1.5+ is recommanded. +## Prerequisite -NOTE: PouchContainer version prior to 0.5.x (including version 0.5.0) did not support configuring CNI network plugin with flannel. If you want to do that, use the latest code from the branch of master, refer to [Developer Quick-Start](https://github.com/alibaba/pouch/blob/master/INSTALLATION.md#developer-quick-start) +Before installing Kubernetes and PouchContainer to provide orchestrating service for users, some environment requirement must be met. -## Install and Configure +- software version requirement +- node resource requirement -An all-in-one kubernetes cluster with PouchContainer runtime could be deployed by running: +### Software Version Requirement -``` -hack/kubernetes/allinone.sh -``` +Running Kubernetes with PouchContainer has several software version restrictions. It may lead to unstable situation if not all of the following conditions are satisfied. + +|Sotware Name|Version| +|:----:|:---:| +|Kubernetes|1.10.0+| +|PouchContainer|1.0.0-rc1+| +|CNI|0.6.0+| +|OS Distribution|CentOS 7.0+ or Ubuntu 16.04 +| +|Linux Kernel|3.10+(CentOS 7.0+) or 4.10+(Ubuntu 16.04+)| + +### Node Resource Requirement + +To make Kubernetes run stably, we should provide sufficient resource for node(s) in Kubernetes. Currently we list two kinds of deploying mode: + +- single node mode(k8s master and kubelet on the same node) +- cluster mode(k8s master and kubelet spreading on multiple nodes) + +Node resource requirement is described as the following: + +|Deploy Mode| CPU Core Requirement| Memory Requirement| +|:-:|:-:|:-:| +|Single Node Mode| 1+ | 2GB +| +|Cluster Mode|1+ (each)| 2GB + (each)| -Please refer to [allinone](https://github.com/alibaba/pouch/blob/master/hack/kubernetes/allinone.sh) . +## Install and Configure PouchContainer + +PouchContainer is a container engine which is a must-install on each node in the cluster. Before making PouchContainer ready, three essential steps are required: + +- install PouchContainer +- configure PouchContainer +- restart PouchContainer + +> NOTE: all commands in the document are executed with ordinary user. However, if super user root can be accessed, the installation works as well. ### Install PouchContainer -You can easily setup a basic PouchContainer environment, see [INSTALLATION.md](../../INSTALLATION.md). +PouchContainer installation could be quite easy by following document [INSTALLATION.md](../../INSTALLATION.md). ### Configure PouchContainer -On Ubuntu 16.04+: +After installing PouchContainer, it will be running by default. However, to make PouchContainer support Kubernetes, configuration of PouchContainer must be updated. In details, only by switching on flag `--enbale-cri` can PouchContainer provide CRI service for kubelet in Kubernetes. So, a sed command can be used to udpate PouchContainer's configuration. -NOTE: If you'd like to use Kubernetes 1.10+, CRI_VERSION should be "v1alpha2" +For Ubuntu 16.04+, the command is as below: +``` shell +sudo sed -i 's/ExecStart=\/usr\/local\/bin\/pouchd/ExecStart=\/usr\/local\/bin\/pouchd --enable-cri=true --cri-version=v1alpha2/g' /lib/systemd/system/pouch.service ``` -CRI_VERSION="v1alpha1" -sed -i 's/ExecStart=\/usr\/bin\/pouchd/ExecStart=\/usr\/bin\/pouchd --enable-cri=true --cri-version=${CRI_VERSION}/g' /usr/lib/systemd/system/pouch.service -systemctl daemon-reload -systemctl restart pouch + +For CentOS 7.0+, the command is as below: + +``` shell +sudo sed -i 's/ExecStart=\/usr\/local\/bin\/pouchd/ExecStart=\/usr\/local\/bin\/pouchd --enable-cri=true --cri-version=v1alpha2/g' /lib/systemd/system/pouch.service ``` -On CentOS 7: +### Restart PouchContainer -NOTE: If you'd like to use Kubernetes 1.10+, CRI_VERSION should be "v1alpha2" +Once the configuration of PouchContainer is updated, restart is a simple and direct way to make the updated configuration work. No matter in Ubuntu 16.04+ or CentOS 7.0+, `systemctl` can be used to manage PouchContainer's restart, as the following: -``` -CRI_VERSION="v1alpha1" -sed -i 's/ExecStart=\/usr\/local\/bin\/pouchd/ExecStart=\/usr\/local\/bin\/pouchd --enable-cri=true --cri-version=${CRI_VERSION}/g' /lib/systemd/system/pouch.service -systemctl daemon-reload -systemctl restart pouch +```shell +sudo systemctl daemon-reload +sudo systemctl restart pouch ``` -### Setup Repo +> NOTE: When enabling CRI in PouchContainer, long running process pouchd will listen to an additional port 10011 by default. Please make sure port 10011 is not blocked by iptables or some other firewall tools. -On Ubuntu 16.04+: +### Verify PouchContainer Correctness -``` -apt-get update && apt-get install -y apt-transport-https -curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - -cat < /etc/apt/sources.list.d/kubernetes.list +After installing and configuring PouchContainer to make it ready for Kubernetes, validation work could be made to verify if the pouchd meets the requirement. + +User can execute a command `pouch info` on the node to see if the field `IsCRIEnabled`in the result is true. If `IsCRIEnabled` is true, it means it is ready for Kubernetes. Otherwise, it is not. + +## Install and Configure Kubernetes + +No matter running Kubernetes on a single node or in a cluster, PouchContainer is the underlying container engine which should be ready first. After that, Kubernetes components should be installed and configured on corresponding nodes. + +Kubernetes part also includes three parts: + +- Download Kubernetes Packages +- Configure Kubernetes Component; +- Initialize Kubernetes Master Node; + +### Download Kubernetes Packages + +When installing Kubernetes, we use [kubeadm](https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm/) to create Kubernetes cluster. First step is to download essential packages from software repository. Thus, we need to add kubernetes.io to debian package source for Ubuntu series and also for CentOS series. After setting up source repository, three Kubernetes components will be downloaded: + +- [kubelet](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/): a primary node agent runs on each node in Kubernetes clusters. +- [kubeadm](https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm/): a tool built to provide best practise for creating Kubernetes clusters. +- [kubectl](https://kubernetes.io/docs/reference/kubectl/overview/): a command line interface for running commands against Kubernetes clusters. + +For Ubuntu 16.04+, execute the following commands: + +``` shell +$ sudo apt-get update && sudo apt-get install -y apt-transport-https +$ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - +$ sudo cat < /etc/apt/sources.list.d/kubernetes.list deb http://apt.kubernetes.io/ kubernetes-xenial main EOF -apt-get update +$ sudo apt-get update +$ export RELEASE="1.10.2-00" +$ apt-get -y install kubelet=${RELEASE} kubeadm=${RELEASE} kubectl=${RELEASE} ``` -On CentOS 7: +For CentOS 7.0+, execute the following commands: -``` -cat < /etc/yum.repos.d/kubernetes.repo +``` shell +$ sudo cat < /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=http://yum.kubernetes.io/repos/kubernetes-el7-x86_64 @@ -91,111 +152,129 @@ repo_gpgcheck=1 gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg EOF +$ export RELEASE="1.10.2-0.x86_64" +$ sudo yum install -y kubelet-${RELEASE} kubeadm-${RELEASE} kubectl-${RELEASE} ``` -### Install Kubernetes Components +### Configure Kubernetes Components -On Ubuntu 16.04+: +After downloading all essential packages, there are some configurations which needs update. For kubelet, configure it to choose PouchContainer as its container runtime. Since PouchContainer makes use of a UNIX socket `unix:///var/run/pouchcri.sock`, this socket path must be delivered to kubelet. Update command is exactly the following one: -```sh -RELEASE="1.9.4-00" -apt-get -y install kubelet=${RELEASE} kubeadm=${RELEASE} kubectl=${RELEASE} +``` shell +sudo sed -i '2 i\Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --container-runtime-endpoint=unix:///var/run/pouchcri.sock --image-service-endpoint=unix:///var/run/pouchcri.sock"' /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +sudo systemctl daemon-reload ``` -On CentOS 7: +### Setup Kubernetes Cluster -```sh -RELEASE="1.9.4-0.x86_64" -yum install -y kubelet-${RELEASE} kubeadm-${RELEASE} kubectl-${RELEASE} -``` +After setting up kubelet's configuration, the next step is installing Kubernetes master node. When using kubeadm to initialize master node, a CIDR address should be provided. CIDR address is an allocated IP range dedicated for pod's network. Please do not make this IP range conflict or overlapped with the undelying network environment. -Configure kubelet with PouchContainer as its runtime: +> NOTE: `--pod-network-cidr 10.244.0.0/16` is required if you need to setup CNI network with flannel and calico later. -```sh -sed -i '2 i\Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --container-runtime-endpoint=unix:///var/run/pouchcri.sock --image-service-endpoint=unix:///var/run/pouchcri.sock"' /etc/systemd/system/kubelet.service.d/10-kubeadm.conf -systemctl daemon-reload +``` shell +sudo kubeadm init --pod-network-cidr 10.244.0.0/16 --ignore-preflight-errors=all ``` -For more details, please check [install kubelet](https://kubernetes.io/docs/setup/independent/install-kubeadm/#installing-kubeadm-kubelet-and-kubectl). - -### Install CNI +After executing the command above, Kubernetes master and kubelet are both running on this node. As a result, **a complete Kubernetes cluster on a singe node has been setup.** -On Ubuntu 16.04+: +Before experiencing service provided by Kubernetes, user must execute three commands **on master node** to make it: -``` -apt-get install -y kubernetes-cni +``` shell +mkdir -p ~/.kube +sudo cp -i /etc/kubernetes/admin.conf ~/.kube/config +sudo chown $(id -u):$(id -g) ~/.kube/config ``` -On CentOS 7: +In addition, in the following information from the command execution output, there will be(note: in your environment $token ${master_ip:port} $ca-cert will be detailed content, just copy this command): ``` -setenforce 0 -yum install -y kubernetes-cni + kubeadm join --token $token ${master_ip:port} --discovery-token-ca-cert-hash $ca-cert ``` -### Setting up the master node +The above command `kubeadm join` is used by any node except master node which wishes to join Kubernete master node as a role of slave. Execute it on all nodes who are playing slave role in cluster. -For more detailed Kubernetes cluster installation, please check [Using kubeadm to Create a Cluster](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/) +> NOTE: Kubernetes installation part does not include the CNI network part. CNI network setup is an individual part, refer [Setup CNI network](). -``` -kubeadm init --pod-network-cidr 10.244.0.0/16 --ignore-preflight-errors=all -``` +### Verify Kubernetes Correctness -Set the KUBECONFIG environment variable +After installing PouchContainer and Kubernetes, user could create a running pod to verify the correctness of cluster. The following example creates a very simple pod which contains only one busybox container. If we see the pod is running, it means that you succeed in deploying Kubernetes with PouchContainer. -```sh -export KUBECONFIG=/etc/kubernetes/admin.conf +``` shell +$ cat busybox.yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: example +spec: + replicas: 1 + selector: + matchLabels: + pouch: busybox + template: + metadata: + labels: + pouch: busybox + spec: + containers: + - name: busyboxx + image: docker.io/library/busybox:latest + command: + - top + hostNetwork: true + +$ sudo kubectl apply -f busybox.yaml +deployment.apps "example" created + +$ sudo kubectl get pods -o wide +NAME READY STATUS RESTARTS AGE IP NODE +example-96b47ff48-45j2f 1/1 Running 0 5m 10.140.0.2 master ``` -Configure CNI network plugin with [flannel](https://github.com/coreos/flannel) +## Setup CNI network -``` -kubectl create -f https://github.com/coreos/flannel/raw/master/Documentation/kube-flannel.yml -``` - -NOTE: For other plugins, please check [Installing a pod network](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#pod-network). +Network in Kubernetes cluster is very decoupled work when deploying Kubernetes. In kubernetes community, CNI plugin is a wide-spread way to setup network for pods. Fully test CNI plugins in Kubernetes community with PouchContainer are: -Optional: enable schedule pods on the master node +- [flannel](https://github.com/coreos/flannel) +- [calico](https://github.com/projectcalico/calico) -```sh -kubectl taint nodes --all node-role.kubernetes.io/master:NoSchedule- -``` +Before setting up CNI network, we recommand reading reference [Install a pod network add-on](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#pod-network) first. -### Setting up the minion nodes +### Flannel -After initializing the master node, you may get the following prompt: +Set `/proc/sys/net/bridge/bridge-nf-call-iptables` to 1 by running `sysctl net.bridge.bridge-nf-call-iptables=1` to pass bridged IPv4 traffic to iptables' chains. It is quite simple for users to setup CNI network with flannel. Only one command is required to execute on master node: +``` shell +kubectl create -f https://github.com/coreos/flannel/raw/master/Documentation/kube-flannel.yml ``` -You can now join any number of machines by running the following on each node -as root: - kubeadm join --token $token ${master_ip:port} --discovery-token-ca-cert-hash $ca-cert -``` +If your environment cannot access Internet, please make sure to store content of file above to local directory. Then `kubectl create -f kube-flannel.yml` can make it setup flannel network. -NOTE: Because kubeadm still assumes docker as the only container runtime ,Use the flag `--ignore-preflight-errors=all` to skip the check. +### Calico -Copy & Run it in all your minion nodes. +Currently Calico works on `amd64` only. It is quite simple for users to setup CNI network with Calico. Only one command is required to execute on master node: -## Run and Verify +``` shell +kubectl apply -f https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml +kubectl apply -f https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml +``` -Create a deployment named `Pouch`: +### Verify Correctness of Network -```sh -# cat pouch.yaml +``` shell +$ cat pouch.yaml apiVersion: apps/v1 kind: Deployment metadata: name: pouch - labels: - pouch: pouch spec: + replicas: 1 selector: matchLabels: - pouch: pouch + pouch: nginx template: metadata: labels: - pouch: pouch + pouch: nginx spec: containers: - name: pouch @@ -203,17 +282,14 @@ spec: ports: - containerPort: 80 -# kubectl create -f pouch.yaml +$ kubectl create -f pouch.yaml deployment "pouch" created -``` - -Confirm the pod of deployment is really running: - -```sh -# kubectl get pods -o wide +$ +$ kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE pouch-7dcd875d69-gq5r9 1/1 Running 0 44m 10.244.1.4 master -# curl 10.244.1.4 +$ +$ curl 10.244.1.4 @@ -241,14 +317,4 @@ Commercial support is available at ``` -## Troubleshooting - -- Because `kubeadm` still assumes docker as the only container runtime which can be used with kubernetes. When you use `kubeadm` to initialize the master node or join the minion node to the cluster, you may encounter the following error message:`[ERROR SystemVerification]: failed to get docker info: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?`. Use the flag `--ignore-preflight-errors=all` to skip the check, like `kubeadm init --ignore-preflight-errors=all`. - -- By default PouchContainer will support CRI v1alpha2,which means that using a version of Kubernetes prior to 1.10 will not work. As the NOTE mentioned above, we could start pouchd with the configuration like `pouchd --cri-version v1alpha1` to specify the version of CRI to support the version of Kubernetes below 1.10. - -- By default PouchContainer will not enable the CRI. If you'd like to deploy Kubernetes with PouchContainer, you should start pouchd with the configuration like `pouchd --enable-cri`. - -- By default PouchContainer will use `registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0` as the image of infra container. If you'd like use image other than that, you could start pouchd with the configuration like `pouchd --enable-cri --sandbox-image XXX`. - -- Any other troubles? Make an issue to connect with us! +When we see the output of nginx, congratulations to succeed in running Kubernetes Clusters with CNI network enabled. diff --git a/docs/kubernetes/troubleshooting.md b/docs/kubernetes/troubleshooting.md new file mode 100644 index 000000000..472cdd8db --- /dev/null +++ b/docs/kubernetes/troubleshooting.md @@ -0,0 +1,11 @@ +# Troubleshooting + +- Because `kubeadm` still assumes docker as the only container runtime which can be used with kubernetes. When you use `kubeadm` to initialize the master node or join the minion node to the cluster, you may encounter the following error message:`[ERROR SystemVerification]: failed to get docker info: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?`. Use the flag `--ignore-preflight-errors=all` to skip the check, like `kubeadm init --ignore-preflight-errors=all`. + +- By default PouchContainer will support CRI v1alpha2,which means that using a version of Kubernetes prior to 1.10 will not work. As the NOTE mentioned above, we could start pouchd with the configuration like `pouchd --cri-version v1alpha1` to specify the version of CRI to support the version of Kubernetes below 1.10. + +- By default PouchContainer will not enable the CRI. If you'd like to deploy Kubernetes with PouchContainer, you should start pouchd with the configuration like `pouchd --enable-cri`. + +- By default PouchContainer will use `registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0` as the image of infra container. If you'd like use image other than that, you could start pouchd with the configuration like `pouchd --enable-cri --sandbox-image XXX`. + +- Any other troubles? Make an issue to connect with us! \ No newline at end of file