Kubespray wrapper

This tool helps to deploy a kubernetes cluster with ansible.

Example on GCE:

Requirements

• Ansible v2.3 (or newer)
• The current user must have its ssh public key installed on the remote servers.
• The remote user (option --user) must be in the sudoers with no password

Installation

Python pip

sudo pip2 install kubespray

Config file

A config file can be updated (yaml). (default: ~/.kubespray.yml )
This file contains default values for some parameters that don't change frequently
Note these values are overwritten by the command line.

# Common options
# ---------------
# Path where the kubespray ansible playbooks will be installed
# Defaults to current user's home directory if not set
# kubespray_path: "/tmp"
# Default inventory path
kubespray_git_repo: "https://github.com/kubernetes-incubator/kubespray.git"
# Logging options
loglevel: "info"

# Amazon web services options
# ---
# aws_access_key: "<key>"
# aws_secret_key: "<secret_key>"
# key_name: "<keypair_name>"
# ami: "<aws_ami>"
# masters_instance_type: "<masters_instance_type>"
# nodes_instance_type: "<nodes_instance_type>"
# etcds_instance_type: "<etcds_instance_type>"
# masters_instance_profile_name: "<master_instance_profile_name>"
# nodes_instance_profile_name: "<nodes_instance_profile_name>"
# etcds_instance_profile_name: "<etcds_instance_profile_name>"
# security_group_name: "<security_group_name>"
# security_group_id: "<security_group_id>"
# assign_public_ip: True
# vpc_subnet_id: "<vpc_id>"
# region: "<aws_region>"
# tags:
#   - type: k8s

# OpenStack options
# ---
# os_auth_url: "https://"
# os_username: "k8s"
# os_password: "secret"
# os_project_name: "k8s"
# masters_flavor: "m1.small"
# nodes_flavor: "m1.small"
# etcds_flavor: "m1.small"
# image: "wily-server-cloudimg-amd64"
# network: "my-network"
# sshkey: "my-pub-key"
...

Basic usage

Note: You may want to choose the architecture of your cluster.
Here are 3 examples:

• 3 vms, all 3 have etcd installed, all 3 are nodes (running pods), 2 of them run master components

kubespray [prepare|aws|gce] --nodes 3

• 6 vms, 2 are nodes and masters, 1 is node only and a distinct etcd cluster

kubespray [prepare|aws|gce] --nodes 3 --etcds 3

• 8 vms, 2 distinct masters, 3 nodes and 3 etcds

kubespray [prepare|aws|gce] --nodes 3 --etcds 3 --masters 2

You should have at least 3 nodes but you can spawn only one instance for tests purposes.

Generate inventory for a baremetal cluster

If the servers are already available you can use the argument prepare The command below will just clone the git repository and creates the inventory. The hostvars must be separated by a comma without spaces

kubespray prepare --nodes node1[ansible_ssh_host=10.99.21.1] node2[ansible_ssh_host=10.99.21.2] node3[ansible_ssh_host=10.99.21.3] [--etcds N+] [--masters N+]

Run instances and generate the inventory on Clouds

AWS

In order to create vms on AWS you can either edit the config file ~/.kubespray.yml or set the options with the argument aws if the config file is filled with the proper information you just need to run the following command

kubespray aws --nodes 3 [--etcds N+] [masters N+] [--nodes-instance-type m4.large]

Another example which download kubespray's repo in a defined directory and set the cluster name

kubespray aws --nodes 3 -p /tmp/mykubespray --cluster-name foobar

GCE

In order to create vms on GCE you can either edit the config file /etc/kubespray/kubespray.yml or set the options with the argument gce if the config file is filled with the proper information you just need to run the following command

kubespray gce --nodes 3

Another example if you already have a kubespray repository in your home dir

kubespray gce --nodes 3 --noclone --cluster-name foobar [--nodes-machine-type n1-standard-4]

OpenStack

In order to create vms on a OpenStack cluster you can either edit the config file ~/.kubespray.yml or set the options with the argument openstack. The options network and sshkey are required and need to be created before running kubespray, you can either create them using the OpenStack Dashboard or the OpenStack CLI clients. Running the ansible routines that kubespray will invoke requires to have installed the openstack cli tools and openstack shade.

Create a network using the OpenStack Neutron client

# create network
$ neutron net-create k8s-network

Created a new network:
+-----------------+--------------------------------------+
| Field           | Value                                |
+-----------------+--------------------------------------+
| admin_state_up  | True                                 |
| id              | 18989cb2-d9fc-4abd-85e8-fe5b33df3541 |
| mtu             | 0                                    |
| name            | k8s-network                          |
| router:external | False                                |
| shared          | False                                |
| status          | ACTIVE                               |
| subnets         |                                      |
| tenant_id       | 5de2cf232a674b05983b61fdc1ea67aa     |
+-----------------+--------------------------------------+

# create subnet
$ neutron subnet-create --name k8s-subnet --dns-nameserver 8.8.8.8 --enable-dhcp --allocation_pool "start=192.168.0.100,end=192.168.0.200" k8s-network 192.168.0.0/24

Created a new subnet:
+-------------------+----------------------------------------------------+
| Field             | Value                                              |
+-------------------+----------------------------------------------------+
| allocation_pools  | {"start": "192.168.0.100", "end": "192.168.0.200"} |
| cidr              | 192.168.0.0/24                                     |
| dns_nameservers   |                                                    |
| enable_dhcp       | True                                               |
| gateway_ip        | 192.168.0.1                                        |
| host_routes       |                                                    |
| id                | fc5a5436-de3a-4d68-9846-d977d047d6d7               |
| ip_version        | 4                                                  |
| ipv6_address_mode |                                                    |
| ipv6_ra_mode      |                                                    |
| name              | k8s-subnet                                         |
| network_id        | 18989cb2-d9fc-4abd-85e8-fe5b33df3541               |
| subnetpool_id     |                                                    |
| tenant_id         | 5de2cf232a674b05983b61fdc1ea67aa                   |
+-------------------+----------------------------------------------------+

# create a router
$ neutron router-create k8s-router

Created a new router:
+-----------------------+--------------------------------------+
| Field                 | Value                                |
+-----------------------+--------------------------------------+
| admin_state_up        | True                                 |
| external_gateway_info |                                      |
| id                    | 69027fa4-239b-4bae-89bc-3040c365ee4d |
| name                  | k8s-router                           |
| routes                |                                      |
| status                | ACTIVE                               |
| tenant_id             | 5de2cf232a674b05983b61fdc1ea67aa     |
+-----------------------+--------------------------------------+

# set gateway network, external_network is the name of the external network defined by your OpenStack provider
$ neutron router-gateway-set k8s-router external_network

Set gateway for router k8s-router

# add k8s-subnet to k8s-router, plug the virtual cable
$ neutron router-interface-add k8s-router subnet=k8s-subnet

Added interface a62ca3b4-3302-45b6-803e-f87a4f43b4b6 to router k8s-router.

Upload a ssh key in order to connect to the vms using the OpenStack Nova client

# import SSH Public Key with name k8s-pub-key
$ nova keypair-add --pub-key id_rsa.pub k8s-pub-key

Once the preparation have been completed you can enter the required options to the config file /etc/kubespray/kubespray.yml:

...

network: "k8s-network"
sshkey: "k8s-pub-key"

...

If the config file is filled with the proper information you just need to run the following command

kubespray openstack --masters 2 --nodes 2 --etcds 3

Another example if you already have a kubespray repository in your home dir

kubespray openstack --masters 2 --nodes 2 --etcds 3 --noclone --cluster-name foobar

Add a node to an existing cluster It's possible to add nodes to a running cluster,
these newly added nodes will act as node only (no etcd, no master components)

Add a node

kubespray [aws|gce] --add --nodes 1

Then deploy the cluster with the same options as the running cluster.

Deploy cluster

The last step is to run the cluster deployment.

Note:

• default network plugin : flannel (vxlan) default
• default kubespray_path : "/home/<current_user>/kubespray"
• inventory path : "<kubespray_path>/inventory/inventory.cfg".
• The option --inventory allows to use an existing inventory (file or dynamic)
• On coreos (--coreos) the directory /opt/bin must be writable
• You can use all Ansible's variables with --ansible-opts '-e foo=bar -e titi=toto -vvv' (the value must be enclosed by simple quotes)

some examples:

Deploy with the default options on baremetal

kubespray deploy

Deploy on AWS using a specific kubespray directory and set the api password

kubespray deploy --aws --passwd secret -p /tmp/mykubespray -n weave

Deploy a kubernetes cluster on CoreOS servers located on GCE

kubespray deploy -u core -p /kubespray-dc1 --gce --coreos --cluster-name mykube --kube-network 10.42.0.0/16