diff --git a/CAPZ-sysext/.gitignore b/CAPZ-sysext/.gitignore new file mode 100644 index 0000000..4b32c8e --- /dev/null +++ b/CAPZ-sysext/.gitignore @@ -0,0 +1,3 @@ +*.swp +demo +azure.env diff --git a/CAPZ-sysext/README.md b/CAPZ-sysext/README.md new file mode 100644 index 0000000..18205f4 --- /dev/null +++ b/CAPZ-sysext/README.md @@ -0,0 +1,219 @@ +# Cluster API Azure (CAPZ) with Flatcar + + +This demo is divided into three sections: +* Automated, scripted set-up of a ClusterAPI worker cluster on Azure, including live in-place updates +* Full manual walk-through of setting up [Cluster API Azure using Flatcar sysext template](#manual-cluster-api-azure-using-flatcar-sysext-template) +* Full manual walk-through of setting up Cluster API Azure using AKS (mixing Ubuntu and Flatcar nodes) (TBD) + +## Automated set-up of a Cluster API demo cluster on Azure + +The automation will lead you through all steps necessary for creating an Azure Cluster API cluster from scratch. +It will also take care of creating a local management cluster, and generally requires very few prerequisites. +The automation is best suitable if you want to start from scratch, and/or if you need a quick throw-away CAPZ cluster. + +**tl;dr** + +```bash +git clone https://github.com/flatcar/flatcar-demos.git +cd flatcar-demos/CAPZ-sysext +mkdir demo +cd demo + +cp ../cluster-template-flatcar-sysext.yaml . +cp ../azure.env.template azure.env +vim azure.env # fill in account information + +bash -i +source ../capz-demo.env + +get_prerequisites +setup_kind_cluster +generate_capz_yaml + +deploy_capz_cluster + +kc_worker get nodes -o wide +``` + +After worker nodes are operational +```bash +kc_worker apply -f kured-dockerhub.yaml +``` + +Watch the in-place update +```bash +watch 'source ../capz-demo.env; kc_worker get nodes -o wide;' +``` + +### Prerequisites + +An Azure account and an active subscription. +We will create an RBAC account for use by the Cluster API automation, so you'll need access to the Active Directory of your account. + +### Prepare the demo + +1. Clone the flatcar-demos repo and change into the "CAPZ-sysext" subdirectory of the repo. + ```bash + git clone https://github.com/flatcar/flatcar-demos.git + cd flatcar-demos/CAPZ-sysext + ``` +2. Create a sub-directory `demo` to work in (will be ignored as per the repo's `.gitgnore`). + ```bash + mkdir demo + cd demo + ``` +3. Copy the upper directory's `cluster-template-flatcar-sysext.yaml`, and copy `azure.env.template` to `azure.env`: + ```bash + cp ../cluster-template-flatcar-sysext.yaml . + cp ../azure.env.template azure.env + ``` +4. Edit `azure.env` and fill in the required variables. + Comments in the file will guide you through creating an Active Directory RBAC account to use for the demo if you don't have one available. + You can also optionally configure an SSH key to log in to the worker cluster's control plane node. + +You should now have: +- the automation available locally from the flatcar-demos repository you've cloned +- a `demo` sub-directory with two files in it: + - `cluster-template-flatcar-sysext.yaml` which the automation will use to generate the worker cluster configuration from + - `azure.env` with data to access the Azure account you want to use for the demo + +### Run the demo + +The demo will first create a local KIND cluster and install the Cluster API Azure provider. +It will then generate a cluster configuration for the workload cluster on Azure, and apply it. +It will wait for the Azure cluster to come up, then give control back to you. + +You can interact with the KIND management cluster by using `kc_mgmt `, and with the workload cluster via `kc_worker `. +`kc_mgmt` and `kc_worker` are just wrappers around kubectl with the respective `kubeconfig` set. + +1. Start a new shell and source the automation + ```bash + bash -i + source ../capz-demo.env + ``` +2. Check prerequisites. This will download `kind` and `helm` for local use. + ```bash + get_prerequisites + ``` +3. Set up the KIND (Kubernetes-IN-Docker") cluster for local use. + ```bash + setup_kind_cluster + ``` +4. Install the Azure provider and generate the Cluster API Azure cluster configuration. + ```bash + generate_capz_yaml + ``` +5. Provision the ClusterAPI Azure workload cluster. + This will apply the cluster configuration to the management cluster, which will start the provisioning process on Azure. + It will then wait until the cluster is fully provisioned, and install cluster add-ons (Calico and the external cloud provider) + to make the cluster operational. + Provisioning the cluster can take some time. + You can watch the resources being created in the `flatcar-capi-demo-azure` resource group used by the automation in the Azure portal. + ```bash + deploy_capz_cluster + ``` + +You can now interact with the workload cluster using `kc_worker`, a simple wrapper around `kubectl`. + +#### Live in-place Kubernetes updates + +The `cluster-template-flatcar-sysext.yaml` shipped with this repo has in-place Kubernetes updates enabled via `systemd-sysupdate`. +An update should already have been staged on the workload cluster (happens almost immediately after provisioning). +The `systemd-sysupdate` configuration will also have created a flag file on the node to signal a need for reboot: `/run/reboot-required`. + +To demo live in-place Kubernetes updates, all we need to do is to provision KureD to the workload cluster. +A suitable kured configuration is shipped with the repository. + +Run +```bash +kc_worker apply -f ../kured-dockerhub.yaml +``` +to start the process. + +KureD will detect the flag file and evacuate and reboot the nodes, one after another. +You can watch the process: +```bash +watch 'source ../capz-demo.env; kc_worker get nodes;' +``` + + +## Manual Cluster API Azure using Flatcar sysext template + +In this demo, you will learn how to create a Kubernetes cluster using Azure resources and powered by Flatcar nodes using the systemd-sysext approach. This is inspired from: https://capz.sigs.k8s.io/getting-started + +### Requirements + +:warning: This is done on a fresh Azure account for demo purposes to avoid interfering with any existing components + +* Azure account with an Azure Service Principal +* A management cluster (e.g any existing Kubernetes cluster) +* `clusterctl` and `yq` up-to-date and available in the `$PATH` + +Contrary to the automated set-up, which creates a local KIND cluster for management, the below assumes you already run a management cluster. + +### Initialize the management cluster + +We first need to export some variables and create some secrets before initializing the management cluster: +```bash +export AZURE_SUBSCRIPTION_ID=a77585be-... +export EXP_KUBEADM_BOOTSTRAP_FORMAT_IGNITION=true +export AZURE_TENANT_ID="" +export AZURE_CLIENT_ID="" +export AZURE_CLIENT_ID_USER_ASSIGNED_IDENTITY=$AZURE_CLIENT_ID # for compatibility with CAPZ v1.16 templates +export AZURE_CLIENT_SECRET="" +export AZURE_RESOURCE_GROUP="capz-demo" +export AZURE_LOCATION="centralus" +``` + +From now, you can just copy-paste: +```bash +# Settings needed for AzureClusterIdentity used by the AzureCluster +export AZURE_CLUSTER_IDENTITY_SECRET_NAME="cluster-identity-secret" +export CLUSTER_IDENTITY_NAME="cluster-identity" +export AZURE_CLUSTER_IDENTITY_SECRET_NAMESPACE="default" + +# Create a secret to include the password of the Service Principal identity created in Azure +# This secret will be referenced by the AzureClusterIdentity used by the AzureCluster +kubectl create secret generic "${AZURE_CLUSTER_IDENTITY_SECRET_NAME}" --from-literal=clientSecret="${AZURE_CLIENT_SECRET}" --namespace "${AZURE_CLUSTER_IDENTITY_SECRET_NAMESPACE}" + +# Finally, initialize the management cluster +clusterctl init --infrastructure azure +``` + +### Create the workload cluster + +Now, you can generate the workload cluster configuration: + +_Notes_: +* at this time, the CAPZ Flatcar sysext PR is still opened (https://github.com/kubernetes-sigs/cluster-api-provider-azure/pull/4575) which means that `--infrastructure azure --flavor flatcar-sysext` must be replaced by `--from /path/to/flatcar-sysext/template.yaml` +* Kubernetes version must match sysext-bakery [releases](https://github.com/flatcar/sysext-bakery/releases/tag/latest) + +```bash +clusterctl generate cluster ${AZURE_RESOURCE_GROUP} \ + --infrastructure azure \ + --kubernetes-version v1.31.1 \ + --control-plane-machine-count=3 \ + --worker-machine-count=3 \ + --flavor flatcar-sysext \ + > "${AZURE_RESOURCE_GROUP}.yaml" +yq -i "with(. | select(.kind == \"AzureClusterIdentity\"); .spec.type |= \"ServicePrincipal\" | .spec.clientSecret.name |= \"${AZURE_CLUSTER_IDENTITY_SECRET_NAME}\" | .spec.clientSecret.namespace |= \"${AZURE_CLUSTER_IDENTITY_SECRET_NAMESPACE}\")" "${AZURE_RESOURCE_GROUP}.yaml" +kubectl apply -f "${AZURE_RESOURCE_GROUP}.yaml" +``` + +After a few minutes, the cluster should be available using latest Flatcar version available on the Azure gallery. + +```bash +clusterctl get kubeconfig "${AZURE_RESOURCE_GROUP}" > "${AZURE_RESOURCE_GROUP}.kubeconfig" +kubectl --kubeconfig "${AZURE_RESOURCE_GROUP}.kubeconfig" get nodes -o wide +``` + +Of course, the nodes will not be ready while CNI and CCM are not deployed, here's a simple example using Calico: +``` +# CNI +export IPV4_CIDR_BLOCK=$(kubectl get cluster "${AZURE_RESOURCE_GROUP}" -o=jsonpath='{.spec.clusterNetwork.pods.cidrBlocks[0]}') +helm repo add projectcalico https://docs.tigera.io/calico/charts && \ +helm install calico projectcalico/tigera-operator --version v3.26.1 -f https://raw.githubusercontent.com/kubernetes-sigs/cluster-api-provider-azure/main/templates/addons/calico/values.yaml --set-string "installation.calicoNetwork.ipPools[0].cidr=${IPV4_CIDR_BLOCK}" --namespace tigera-operator --create-namespace +# CCM +helm install --repo https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/helm/repo cloud-provider-azure --generate-name --set infra.clusterName=${AZURE_RESOURCE_GROUP} --set "cloudControllerManager.clusterCIDR=${IPV4_CIDR_BLOCK}" --set-string "cloudControllerManager.caCertDir=/usr/share/ca-certificates" +``` diff --git a/CAPZ-sysext/azure.env.template b/CAPZ-sysext/azure.env.template new file mode 100644 index 0000000..9822b44 --- /dev/null +++ b/CAPZ-sysext/azure.env.template @@ -0,0 +1,28 @@ +# Template for Azure CAPZ settings + +# The subscrition ID to use for the workload cluster. +export AZURE_SUBSCRIPTION_ID="TODO add subscription ID" + +# From https://capz.sigs.k8s.io/getting-started: +# az ad sp create-for-rbac --role contributor --scopes="/subscriptions/${AZURE_SUBSCRIPTION_ID}" +export AZURE_TENANT_ID="TODO add 'tenant' from output of az command" +export AZURE_CLIENT_ID="TODO add 'appId from output of az command'" +export AZURE_CLIENT_SECRET="TODO add 'password' from output of az command" + +# Uncomment and set this to the base64 encoded public component +# of the SSH key you want to use to log into the control plane node, e.g.: +# base64 -w0 ~/.ssh/id_rsa.pub +# Leave commented out if you don't need SSH access. +#export AZURE_SSH_PUBLIC_KEY_B64="" + + +# +# These usually do not need to be touched +# + +export AZURE_CLIENT_ID_USER_ASSIGNED_IDENTITY="${AZURE_CLIENT_ID}" # for compatibility with CAPZ v1.16 templates + +# AZURE_RESOURCE_GROUP is set in capz-demo.env +export AZURE_CLUSTER_IDENTITY_SECRET_NAME="${AZURE_RESOURCE_GROUP}-cluster-identity-secret" +export CLUSTER_IDENTITY_NAME="${AZURE_RESOURCE_GROUP}-cluster-identity" +export AZURE_CLUSTER_IDENTITY_SECRET_NAMESPACE="default" diff --git a/CAPZ-sysext/capz-demo.env b/CAPZ-sysext/capz-demo.env new file mode 100644 index 0000000..d26d983 --- /dev/null +++ b/CAPZ-sysext/capz-demo.env @@ -0,0 +1,232 @@ +#!/bin/bash + +KUBERNETES_VERSION="v1.30.0" + +AZURE_RESOURCE_GROUP="flatcar-capi-demo-azure" +export AZURE_RESOURCE_GROUP + +WORKER_CONTROLPLANE_NODES=1 +WORKER_NODES=2 + +p() { + echo + echo "#####################################" + echo -e "${@}" + echo "-------------------------------------" +} +# -- + +check_command() { + local cmd="$1" + + if ! command -v "$cmd" &> /dev/null ; then + echo "'$cmd' could not be found. Please install your distro's '$cmd'." + return 1 + fi + echo " - '$cmd'" +} +# -- + +check_file() { + local f="$1" + + if [ ! -f "./$f" ] ; then + echo "prerequisite '$f' could not be found." + return 1 + fi + + echo " - '$f'" +} +# -- + +get_prerequisites() { + p "Prerequisites: Checking for required host commands." + check_command "kubectl" || return + check_command "yq" || return + check_command "wget" || return + + p "Prerequisites: Checking for prerequisite files." + check_file "azure.env" || return + check_file "cluster-template-flatcar-sysext.yaml" || return + + if [ ! -f ./clusterctl ] ; then + p "Prerequisites: fetching clusterctl" + wget https://github.com/kubernetes-sigs/cluster-api/releases/latest/download/clusterctl-linux-amd64 + mv clusterctl-linux-amd64 clusterctl + chmod 755 clusterctl + fi + + if [ ! -f ./kind ] ; then + p "Prerequisites: fetching kind" + wget https://github.com/kubernetes-sigs/kind/releases/latest/download/kind-linux-amd64 + mv kind-linux-amd64 kind + chmod 755 kind + fi + + if [ ! -f ./helm ] ; then + p "Prerequisites: fetching helm" + curl https://get.helm.sh/helm-v3.16.2-linux-amd64.tar.gz \ + | tar xz linux-amd64/helm -O >helm + chmod 755 helm + mkdir -p helm-cache + fi +} +# -- + +setup_kind_cluster() { + p "Bootsstrapping cluster" + ./kind create cluster --kubeconfig=./kind-mgmt.kubeconfig + export EXP_KUBEADM_BOOTSTRAP_FORMAT_IGNITION=true +} +# -- + +kc_mgmt() { + kubectl --kubeconfig=./kind-mgmt.kubeconfig "${@}" + +} +# -- + +kc_worker() { + kubectl --kubeconfig=./${AZURE_RESOURCE_GROUP}.kubeconfig "${@}" +} +# -- + +generate_capz_yaml() { + source ./azure.env + + p "Initialising ClusterAPI Azure provider." + ./clusterctl init --infrastructure azure --kubeconfig=./kind-mgmt.kubeconfig + + # FIXME: add + # --infrastructure azure \ + # --flavor flatcar-sysext \ + # and remove + # --from cluster-template-flatcar-sysext.yaml \ + # when https://github.com/kubernetes-sigs/cluster-api-provider-azure/pull/4575 is merged. + + # TODO: remove when PR is merged + export AZURE_CONTROL_PLANE_MACHINE_TYPE="Standard_D2s_v3" + export AZURE_LOCATION="northeurope" + export AZURE_NODE_MACHINE_TYPE="Standard_D2s_v3" + + export CI_RG="${AZURE_RESOURCE_GROUP}" + export FLATCAR_VERSION="latest" + + p "Generating ${AZURE_RESOURCE_GROUP}.yaml." + ./clusterctl generate cluster ${AZURE_RESOURCE_GROUP} \ + --from cluster-template-flatcar-sysext.yaml \ + --kubeconfig=./kind-mgmt.kubeconfig \ + --kubernetes-version "${KUBERNETES_VERSION}" \ + --control-plane-machine-count=${WORKER_CONTROLPLANE_NODES} \ + --worker-machine-count=${WORKER_NODES} \ + > ${AZURE_RESOURCE_GROUP}.yaml + + yq -i "with(. | select(.kind == \"AzureClusterIdentity\"); .spec.type |= \"ServicePrincipal\" | .spec.clientSecret.name |= \"${AZURE_CLUSTER_IDENTITY_SECRET_NAME}\" | .spec.clientSecret.namespace |= \"${AZURE_CLUSTER_IDENTITY_SECRET_NAMESPACE}\")" "${AZURE_RESOURCE_GROUP}.yaml" +} +# -- + +deploy_capz_cluster() { + + p "Creating client secrets and workload cluster" + + kc_mgmt create secret generic "${AZURE_CLUSTER_IDENTITY_SECRET_NAME}" \ + --from-literal=clientSecret="${AZURE_CLIENT_SECRET}" \ + --namespace "${AZURE_CLUSTER_IDENTITY_SECRET_NAMESPACE}" + + sleep 1 + + kc_mgmt apply -f ./${AZURE_RESOURCE_GROUP}.yaml + + p "Waiting for cluster to be provisioned." + while ! kc_mgmt get cluster | grep "${AZURE_RESOURCE_GROUP}" | grep -i provisioned ; do + sleep 1 + done + + # Hack alert: sometimes ended up with an empty kubeconfig from the command below, so + # we add a defensive sleep. + sleep 2 + + ./clusterctl get kubeconfig ${AZURE_RESOURCE_GROUP} \ + --kubeconfig=./kind-mgmt.kubeconfig \ + --namespace default \ + > ./${AZURE_RESOURCE_GROUP}.kubeconfig + + p "Waiting for all nodes to come up." + local count=0 + local target_count=$((WORKER_CONTROLPLANE_NODES + WORKER_NODES)) + while [ "$count" -lt "$target_count" ] ; do + count="$(kc_worker --request-timeout=5s get nodes \ + | grep "${AZURE_RESOURCE_GROUP}" \ + | wc -l)" + echo "$count of $target_count nodes are up." + done + + local worker_cidr="$(kc_mgmt get cluster "${AZURE_RESOURCE_GROUP}" \ + -o=jsonpath='{.spec.clusterNetwork.pods.cidrBlocks[0]}')" + + p "Deploying Calico to cluster CIDR '$worker_cidr' so worker nodes can talk to each other" + + helm_wrapper() { + ./helm --kubeconfig=./${AZURE_RESOURCE_GROUP}.kubeconfig \ + --registry-config=./${AZURE_RESOURCE_GROUP}.helm-registry.json \ + --repository-cache=./helm-cache \ + --repository-config=./${AZURE_RESOURCE_GROUP}.helm-registry.yaml \ + ${@} + } + + helm_wrapper \ + repo add projectcalico https://docs.tigera.io/calico/charts + + helm_wrapper \ + install calico projectcalico/tigera-operator \ + --version v3.26.1 \ + -f https://raw.githubusercontent.com/kubernetes-sigs/cluster-api-provider-azure/main/templates/addons/calico/values.yaml \ + --set-string "installation.calicoNetwork.ipPools[0].cidr=${worker_cidr}" \ + --namespace tigera-operator \ + --create-namespace + + p "Installing Azure cluster controller" + helm_wrapper \ + install --repo https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/helm/repo \ + cloud-provider-azure \ + --generate-name --set infra.clusterName=${AZURE_RESOURCE_GROUP} \ + --set "cloudControllerManager.clusterCIDR=${worker_cidr}" \ + --set-string "cloudControllerManager.caCertDir=/usr/share/ca-certificates" + + p "Your nodes should be ready soon" + kc_worker get nodes + + local bastion="$(kc_mgmt get azurecluster flatcar-capi-demo-azure -o json \ + | jq '.spec.networkSpec.apiServerLB.frontendIPs[0].publicIP.dnsName')" + + local ssh="ssh capi@${bastion}" + p "Cluster is deployed and can now be used ('kc_worker' kubectl wrapper)." + p "You can use 'az serial-console connect -g ${AZURE_RESOURCE_GROUP} -n ' connects via serial console, or\n" \ + "'${ssh}' to ssh into the control plane node (if you set AZURE_SSH_PUBLIC_KEY_B64 in azure.env)." +} +# -- + +cleanup() { + kc_mgmt delete cluster ${AZURE_RESOURCE_GROUP} + ./kind delete cluster --kubeconfig=./kind-mgmt.kubeconfig +} +# -- + +help() { + cat < /tmp/kubernetes" + ExecStartPre=/usr/lib/systemd/systemd-sysupdate -C kubernetes update + ExecStartPost=/usr/bin/sh -c "readlink --canonicalize /etc/extensions/kubernetes.raw > /tmp/kubernetes-new" + ExecStartPost=/usr/bin/sh -c "if ! cmp --silent /tmp/kubernetes /tmp/kubernetes-new; then touch /run/reboot-required; fi" + - name: update-engine.service + # Set this to 'false' if you want to enable Flatcar auto-update + mask: ${FLATCAR_DISABLE_AUTO_UPDATE:=true} + - name: locksmithd.service + # NOTE: To coordinate the node reboot in this context, we recommend to use Kured. + mask: true + - name: systemd-sysupdate.timer + # Set this to 'true' if you want to enable the Kubernetes auto-update. + # NOTE: Only patches version will be pulled. + enabled: true + dropins: + - name: bootcheck.conf + contents: | + [Timer] + OnBootSec=1min + OnUnitActiveSec=10s + RandomizedDelaySec=1s + - name: kubeadm.service + dropins: + - name: 10-flatcar.conf + contents: | + [Unit] + After=oem-cloudinit.service + # kubeadm must run after containerd - see https://github.com/kubernetes-sigs/image-builder/issues/939. + After=containerd.service + joinConfiguration: + nodeRegistration: + kubeletExtraArgs: + cloud-provider: external + name: '@@HOSTNAME@@' + postKubeadmCommands: [] + preKubeadmCommands: + - sed -i "s/@@HOSTNAME@@/$(curl -s -H Metadata:true --noproxy '*' 'http://169.254.169.254/metadata/instance?api-version=2020-09-01' + | jq -r .compute.name)/g" /etc/kubeadm.yml +--- +apiVersion: cluster.x-k8s.io/v1beta1 +kind: Cluster +metadata: + name: ${CLUSTER_NAME} + namespace: default +spec: + clusterNetwork: + pods: + cidrBlocks: + - 192.168.0.0/16 + controlPlaneRef: + apiVersion: controlplane.cluster.x-k8s.io/v1beta1 + kind: KubeadmControlPlane + name: ${CLUSTER_NAME}-control-plane + infrastructureRef: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: AzureCluster + name: ${CLUSTER_NAME} +--- +apiVersion: cluster.x-k8s.io/v1beta1 +kind: MachineDeployment +metadata: + name: ${CLUSTER_NAME}-md-0 + namespace: default +spec: + clusterName: ${CLUSTER_NAME} + replicas: ${WORKER_MACHINE_COUNT} + selector: + matchLabels: null + template: + spec: + bootstrap: + configRef: + apiVersion: bootstrap.cluster.x-k8s.io/v1beta1 + kind: KubeadmConfigTemplate + name: ${CLUSTER_NAME}-md-0 + clusterName: ${CLUSTER_NAME} + infrastructureRef: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: AzureMachineTemplate + name: ${CLUSTER_NAME}-md-0 + version: ${KUBERNETES_VERSION} +--- +apiVersion: controlplane.cluster.x-k8s.io/v1beta1 +kind: KubeadmControlPlane +metadata: + name: ${CLUSTER_NAME}-control-plane + namespace: default +spec: + kubeadmConfigSpec: + clusterConfiguration: + apiServer: + extraArgs: + cloud-provider: external + timeoutForControlPlane: 20m + controllerManager: + extraArgs: + allocate-node-cidrs: "false" + cloud-provider: external + cluster-name: ${CLUSTER_NAME} + etcd: + local: + dataDir: /var/lib/etcddisk/etcd + extraArgs: + quota-backend-bytes: "8589934592" + diskSetup: + filesystems: + - device: /dev/disk/azure/scsi1/lun0 + extraOpts: + - -E + - lazy_itable_init=1,lazy_journal_init=1 + filesystem: ext4 + label: etcd_disk + overwrite: false + partitions: [] + files: + - contentFrom: + secret: + key: control-plane-azure.json + name: ${CLUSTER_NAME}-control-plane-azure-json + owner: root:root + path: /etc/kubernetes/azure.json + permissions: "0644" + format: ignition + ignition: + containerLinuxConfig: + additionalConfig: | + systemd: + units: + - name: systemd-sysupdate.service + dropins: + - name: kubernetes.conf + contents: | + [Service] + ExecStartPre=/usr/bin/sh -c "readlink --canonicalize /etc/extensions/kubernetes.raw > /tmp/kubernetes" + ExecStartPre=/usr/lib/systemd/systemd-sysupdate -C kubernetes update + ExecStartPost=/usr/bin/sh -c "readlink --canonicalize /etc/extensions/kubernetes.raw > /tmp/kubernetes-new" + ExecStartPost=/usr/bin/sh -c "if ! cmp --silent /tmp/kubernetes /tmp/kubernetes-new; then touch /run/reboot-required; fi" + - name: update-engine.service + # Set this to 'false' if you want to enable Flatcar auto-update + mask: ${FLATCAR_DISABLE_AUTO_UPDATE:=true} + - name: locksmithd.service + # NOTE: To coordinate the node reboot in this context, we recommend to use Kured. + mask: true + - name: systemd-sysupdate.timer + # Set this to 'true' if you want to enable the Kubernetes auto-update. + # NOTE: Only patches version will be pulled. + enabled: true + dropins: + - name: bootcheck.conf + contents: | + [Timer] + OnBootSec=1min + OnUnitActiveSec=10s + RandomizedDelaySec=1s + - name: kubeadm.service + dropins: + - name: 10-flatcar.conf + contents: | + [Unit] + After=oem-cloudinit.service + # kubeadm must run after containerd - see https://github.com/kubernetes-sigs/image-builder/issues/939. + After=containerd.service + # Workaround for https://github.com/kubernetes-sigs/cluster-api/issues/7679. + storage: + disks: + - device: /dev/disk/azure/scsi1/lun0 + partitions: + - number: 1 + links: + - path: /etc/extensions/kubernetes.raw + hard: false + target: /opt/extensions/kubernetes/kubernetes-${KUBERNETES_VERSION}-x86-64.raw + files: + - path: /etc/sysupdate.kubernetes.d/kubernetes-${KUBERNETES_VERSION%.*}.conf + mode: 0644 + contents: + remote: + url: https://github.com/flatcar/sysext-bakery/releases/download/latest/kubernetes-${KUBERNETES_VERSION%.*}.conf + - path: /etc/sysupdate.d/noop.conf + mode: 0644 + contents: + remote: + url: https://github.com/flatcar/sysext-bakery/releases/download/latest/noop.conf + - path: /opt/extensions/kubernetes/kubernetes-${KUBERNETES_VERSION}-x86-64.raw + contents: + remote: + url: https://github.com/flatcar/sysext-bakery/releases/download/latest/kubernetes-${KUBERNETES_VERSION}-x86-64.raw + initConfiguration: + nodeRegistration: + kubeletExtraArgs: + cloud-provider: external + name: '@@HOSTNAME@@' + joinConfiguration: + nodeRegistration: + kubeletExtraArgs: + cloud-provider: external + name: '@@HOSTNAME@@' + mounts: + - - etcd_disk + - /var/lib/etcddisk + postKubeadmCommands: [] + preKubeadmCommands: + - sed -i "s/@@HOSTNAME@@/$(curl -s -H Metadata:true --noproxy '*' 'http://169.254.169.254/metadata/instance?api-version=2020-09-01' + | jq -r .compute.name)/g" /etc/kubeadm.yml + machineTemplate: + infrastructureRef: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: AzureMachineTemplate + name: ${CLUSTER_NAME}-control-plane + replicas: ${CONTROL_PLANE_MACHINE_COUNT:=1} + version: ${KUBERNETES_VERSION} +--- +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: AzureCluster +metadata: + name: ${CLUSTER_NAME} + namespace: default +spec: + identityRef: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: AzureClusterIdentity + name: ${CLUSTER_IDENTITY_NAME} + location: ${AZURE_LOCATION} + networkSpec: + subnets: + - name: control-plane-subnet + role: control-plane + - name: node-subnet + role: node + vnet: + name: ${AZURE_VNET_NAME:=${CLUSTER_NAME}-vnet} + resourceGroup: ${AZURE_RESOURCE_GROUP:=${CLUSTER_NAME}} + subscriptionID: ${AZURE_SUBSCRIPTION_ID} +--- +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: AzureClusterIdentity +metadata: + labels: + clusterctl.cluster.x-k8s.io/move-hierarchy: "true" + name: ${CLUSTER_IDENTITY_NAME} + namespace: default +spec: + allowedNamespaces: {} + clientID: ${AZURE_CLIENT_ID_USER_ASSIGNED_IDENTITY} + tenantID: ${AZURE_TENANT_ID} + type: ${CLUSTER_IDENTITY_TYPE:=WorkloadIdentity} +--- +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: AzureMachineTemplate +metadata: + name: ${CLUSTER_NAME}-control-plane + namespace: default +spec: + template: + spec: + dataDisks: + - diskSizeGB: 256 + lun: 0 + nameSuffix: etcddisk + image: + marketplace: + version: ${FLATCAR_VERSION} + publisher: kinvolk + offer: flatcar-container-linux-corevm-amd64 + sku: stable-gen2 + osDisk: + diskSizeGB: 128 + osType: Linux + sshPublicKey: ${AZURE_SSH_PUBLIC_KEY_B64:=""} + vmSize: ${AZURE_CONTROL_PLANE_MACHINE_TYPE} +--- +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: AzureMachineTemplate +metadata: + name: ${CLUSTER_NAME}-md-0 + namespace: default +spec: + template: + spec: + image: + marketplace: + version: ${FLATCAR_VERSION} + publisher: kinvolk + offer: flatcar-container-linux-corevm-amd64 + sku: stable-gen2 + osDisk: + diskSizeGB: 128 + osType: Linux + sshPublicKey: ${AZURE_SSH_PUBLIC_KEY_B64:=""} + vmSize: ${AZURE_NODE_MACHINE_TYPE}