The Azure Kubernetes Engine (aks-engine
) generates ARM (Azure Resource Manager) templates for Kubernetes clusters on Microsoft Azure. The input to aks-engine is a cluster definition file which describes the desired cluster, including orchestrator, features, and agents. The structure of the input files is very similar to the public API for Azure Kubernetes Service.
The following prerequisites are required for a successful use of AKS Engine.
- An Azure Subscription
- The Azure CLI
Binary downloads for the latest version of aks-engine are available on Github. Download AKS Engine for your operating system, extract the binary and copy it to your $PATH
.
You can also choose to install aks-engine using gofish. To do so, execute the command gofish install aks-engine
. You can install gofish following the instructions for your OS.
On macOS, you can install aks-engine with Homebrew. Run the command brew install Azure/aks-engine/aks-engine
to do so. You can install Homebrew following these instructions.
On Windows, you can install aks-engine via Chocolatey by executing the command choco install aks-engine
. You can install Chocolatey following these instructions.
On Linux, if you prefer, you can install aks-engine via install script doing:
$ curl -o get-akse.sh https://raw.githubusercontent.com/Azure/aks-engine/master/scripts/get-akse.sh
$ chmod 700 get-akse.sh
$ ./get-akse.sh
If you would prefer to build AKS Engine from source, or you are interested in contributing to AKS Engine, see the developer guide for more information.
AKS Engine supports bash completion. To enable this, add the following to your .bashrc
or ~/.profile
source <(aks-engine completion)
aks-engine
reads a cluster definition which describes the size, shape, and configuration of your cluster. This guide takes the default configuration of one master and two Linux agents. If you would like to change the configuration, edit examples/kubernetes.json
before continuing.
The aks-engine deploy
command automates creation of a Service Principal, Resource Group and SSH key for your cluster. If operators need more control or are interested in the individual steps see the "Long Way" section below.
NOTE: AKS Engine creates a cluster; it doesn't create an Azure Kubernetes Service (AKS) resource. Clusters that you create using the aks-engine
command (or ARM templates generated by the aks-engine
command) won't show up as AKS resources, for example when you run az aks list
. Think of aks-engine
as the, err, engine which AKS uses to create clusters: you can use the same engine yourself, but AKS won't know about the results.
After the cluster is deployed, the upgrade and scale commands can be used to make updates to your cluster.
- The subscription in which you would like to provision the cluster. This is a uuid which can be found with
az account list -o table
. - Proper access rights within the subscription; especially the right to create and assign service principals to applications
- A
dnsPrefix
which forms part of the the hostname for your cluster (e.g. staging, prodwest, blueberry). The DNS prefix must be unique so pick a random name. - A location to provision the cluster e.g.
westus2
.
$ az account list -o table
Name CloudName SubscriptionId State IsDefault
----------------------------------------------- ----------- ------------------------------------ ------- -----------
Contoso Subscription AzureCloud 51ac25de-afdg-9201-d923-8d8e8e8e8e8e Enabled True
For this example, the subscription id is 51ac25de-afdg-9201-d923-8d8e8e8e8e8e
, the DNS prefix is contoso-apple
, and the location is westus2
.
First, we need to log in to Azure:
$ az login
Note, we have launched a browser for you to login. For old experience with device code, use "az login --use-device-code"
You have logged in. Now let us find all the subscriptions to which you have access...
Next, we'll create a resource group. A resource group is a container that holds related resources for an Azure solution. In Azure, you logically group related resources such as storage accounts, virtual networks, and virtual machines (VMs) to deploy, manage, and maintain them as a single entity. In this case, we want to deploy, manage and maintain the whole Kubernetes cluster as a single entity.
$ az group create --name contoso-apple --location westus2
{
"id": "/subscriptions/51ac25de-afdg-9201-d923-8d8e8e8e8e8e/resourceGroups/contoso-apple",
"location": "westus2",
"managedBy": null,
"name": "contoso-apple",
"properties": {
"provisioningState": "Succeeded"
},
"tags": null
}
Once that's done, we need to create a service principal for the Kubernetes cluster so it can talk to any resources that are a part of the same resource group.
$ az ad sp create-for-rbac --role="Contributor" --scopes="/subscriptions/51ac25de-afdg-9201-d923-8d8e8e8e8e8e/resourceGroups/contoso-apple"
{
"appId": "47a62f0b-917c-4def-aa85-9b010455e591",
"displayName": "azure-cli-2019-01-11-22-22-06",
"name": "http://azure-cli-2019-01-11-22-22-06",
"password": "26054d2b-799b-448e-962a-783d0d6f976b",
"tenant": "72f988bf-86f1-41af-91ab-2d7cd011db47"
}
Make a note of the appId
and the password
fields, as we will be providing them as parameters in the next step.
Finally, run aks-engine deploy
with the appropriate arguments:
$ aks-engine deploy --subscription-id 51ac25de-afdg-9201-d923-8d8e8e8e8e8e \
--dns-prefix contoso-apple \
--resource-group contoso-apple \
--location westus2 \
--api-model examples/kubernetes.json \
--client-id 47a62f0b-917c-4def-aa85-9b010455e591 \
--client-secret 26054d2b-799b-448e-962a-783d0d6f976b \
--set servicePrincipalProfile.clientId="47a62f0b-917c-4def-aa85-9b010455e591" \
--set servicePrincipalProfile.secret="26054d2b-799b-448e-962a-783d0d6f976b"
WARN[0005] apimodel: missing masterProfile.dnsPrefix will use "contoso-apple"
WARN[0005] --resource-group was not specified. Using the DNS prefix from the apimodel as the resource group name: contoso-apple
INFO[0034] Starting ARM Deployment (contoso-apple-1423145182). This will take some time...
INFO[0393] Finished ARM Deployment (contoso-apple-1423145182).
aks-engine
will output Azure Resource Manager (ARM) templates, SSH keys, and a kubeconfig file in _output/contoso-apple-59769a59
directory:
_output/contoso-apple-59769a59/azureuser_rsa
_output/contoso-apple-59769a59/kubeconfig/kubeconfig.westus2.json
aks-engine generates kubeconfig files for each possible region. Access the new cluster by using the kubeconfig generated for the cluster's location. This example used westus2
, so the kubeconfig is _output/<clustername>/kubeconfig/kubeconfig.westus2.json
:
$ KUBECONFIG=_output/contoso-apple-59769a59/kubeconfig/kubeconfig.westus2.json kubectl cluster-info
Kubernetes master is running at https://contoso-apple-59769a59.westus2.cloudapp.azure.com
Heapster is running at https://contoso-apple-59769a59.westus2.cloudapp.azure.com/api/v1/proxy/namespaces/kube-system/services/heapster
KubeDNS is running at https://contoso-apple-59769a59.westus2.cloudapp.azure.com/api/v1/proxy/namespaces/kube-system/services/kube-dns
kubernetes-dashboard is running at https://contoso-apple-59769a59.westus2.cloudapp.azure.com/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
Administrative note: By default, the directory where aks-engine stores cluster configuration (_output/contoso-apple
above) won't be overwritten as a result of subsequent attempts to deploy a cluster using the same --dns-prefix
) To re-use the same resource group name repeatedly, include the --force-overwrite
command line option with your aks-engine deploy
command. On a related note, include an --auto-suffix
option to append a randomly generated suffix to the dns-prefix to form the resource group name, for example if your workflow requires a common prefix across multiple cluster deployments. Using the --auto-suffix
pattern appends a compressed timestamp to ensure a unique cluster name (and thus ensure that each deployment's configuration artifacts will be stored locally under a discrete _output/<resource-group-name>/
directory).
Note: If the cluster is using an existing VNET, please see the Custom VNET feature documentation for additional steps that must be completed after cluster provisioning.
This example uses the more traditional method of generating raw ARM templates, which are submitted to Azure using the az group deployment create
command.
For this example, we will use the same information as before: the subscription id is 51ac25de-afdg-9201-d923-8d8e8e8e8e8e
, the DNS prefix is contoso-apple
, and the location is westus2
.
Before we do anything, we need to log in to Azure:
$ az login
Note, we have launched a browser for you to login. For old experience with device code, use "az login --use-device-code"
You have logged in. Now let us find all the subscriptions to which you have access...
We will also need to generate an SSH key. When creating VMs, you will need an SSH RSA key for SSH access. Use the following articles to create your SSH RSA Key:
- Windows - https://www.digitalocean.com/community/tutorials/how-to-create-ssh-keys-with-putty-to-connect-to-a-vps
- Mac and Linux - https://help.github.com/articles/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent/
Next, we'll create a resource group. A resource group is a container that holds related resources for an Azure solution. In Azure, you logically group related resources such as storage accounts, virtual networks, and virtual machines (VMs) to deploy, manage, and maintain them as a single entity. In this case, we want to deploy, manage and maintain the whole Kubernetes cluster as a single entity.
$ az group create --name contoso-apple --location westus2
{
"id": "/subscriptions/51ac25de-afdg-9201-d923-8d8e8e8e8e8e/resourceGroups/contoso-apple",
"location": "westus2",
"managedBy": null,
"name": "contoso-apple",
"properties": {
"provisioningState": "Succeeded"
},
"tags": null
}
Once that's done, we need to create a service principal for the Kubernetes cluster so it can talk to the any resources that are a part of the same resource group.
$ az ad sp create-for-rbac --role="Contributor" --scopes="/subscriptions/51ac25de-afdg-9201-d923-8d8e8e8e8e8e/resourceGroups/contoso-apple"
{
"appId": "47a62f0b-917c-4def-aa85-9b010455e591",
"displayName": "azure-cli-2019-01-11-22-22-06",
"name": "http://azure-cli-2019-01-11-22-22-06",
"password": "26054d2b-799b-448e-962a-783d0d6f976b",
"tenant": "72f988bf-86f1-41af-91ab-2d7cd011db47"
}
Make a note of the appId
and the password
fields, as we will be providing them in the next step.
AKS Engine consumes a cluster definition which outlines the desired shape, size, and configuration of Kubernetes. There are a number of features that can be enabled through the cluster definition: check the examples
directory for a number of... examples.
Edit the simple Kubernetes cluster definition and fill out the required values:
dnsPrefix
: must be a region-unique name and will form part of the hostname (e.g. myprod1, staging, leapingllama) - be unique!keyData
: must contain the public portion of the SSH key we generated - this will be associated with theadminUsername
value found in the same section of the cluster definition (e.g. 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABA....')clientId
: this is the service principal's appId uuid or name from earliersecret
: this is the service principal's password or randomly-generated password from earlier
Optional: attach to an existing virtual network (VNET). Details here
The generate command takes a cluster definition and outputs a number of templates which describe your Kubernetes cluster. By default, generate
will create a new directory named after your cluster nested in the _output
directory. If your dnsPrefix was contoso-apple
, your cluster templates would be found in _output/contoso-apple-
.
Run aks-engine generate examples/kubernetes.json
The generate
command lets you override values from the cluster definition file without having to update the file. You can use the --set
flag to do that:
aks-engine generate --set linuxProfile.adminUsername=myNewUsername,masterProfile.count=3 clusterdefinition.json
The --set
flag only supports JSON properties under properties
. You can also work with arrays, like the following:
aks-engine generate --set agentPoolProfiles[0].count=5,agentPoolProfiles[1].name=myPoolName clusterdefinition.json
- To enable the optional network policy enforcement using calico, you have to set the parameter during this step according to this guide
- To enable the optional network policy enforcement using cilium, you have to set the parameter during this step according to this guide
- To enable the optional network policy enforcement using antrea, you have to set the parameter during this step according to this guide
Now we can deploy the files azuredeploy.json
and azuredeploy.parameters.json
using either the Azure CLI or PowerShell.
Using the CLI:
$ az group deployment create \
--name "contoso-apple-k8s" \
--resource-group "contoso-apple" \
--template-file "./_output/contoso-apple-abc123/azuredeploy.json" \
--parameters "./_output/contoso-apple-abc123/azuredeploy.parameters.json"
Note: If the cluster is using an existing VNET, please see the Custom VNET feature documentation for additional steps that must be completed after cluster provisioning.