From 3369c78be29d0d790de645b3d7c44c895936c2d0 Mon Sep 17 00:00:00 2001 From: Mathieu Tortuyaux Date: Mon, 28 Oct 2024 14:54:01 +0100 Subject: [PATCH 1/4] capz-sysext: add regular sysext demo Signed-off-by: Mathieu Tortuyaux --- CAPZ-sysext/README.md | 70 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 70 insertions(+) create mode 100644 CAPZ-sysext/README.md diff --git a/CAPZ-sysext/README.md b/CAPZ-sysext/README.md new file mode 100644 index 0000000..d5a7eed --- /dev/null +++ b/CAPZ-sysext/README.md @@ -0,0 +1,70 @@ +# Cluster API Azure (CAPZ) with Flatcar + +This demo is divided into two sections: +* [Cluster API Azure using Flatcar sysext template](#cluster-api-azure-using-flatcar-sysext-template) +* Cluster API Azure using AKS (mixing Ubuntu and Flatcar nodes) + +## Cluster API Azure using Flatcar sysext template + +In this demo, you will learn how to create a Kubernetes cluster using Azure resources and powered by Flatcar nodes using the systemd-sysext approach. This is inspired from: https://capz.sigs.k8s.io/getting-started + +### Requirements + +:warning: This is done on a fresh Azure account for demo purposes to avoid interfering with any existing components + +* Azure account with an Azure Service Principal +* A management cluster (e.g any existing Kubernetes cluster) +* `clusterctl` and `yq` up-to-date and available in the `$PATH` + +### Initialize the management cluster + +We first need to export some variables and create some secrets before initializing the management cluster: +```bash +export AZURE_SUBSCRIPTION_ID=a77585be-... +export EXP_KUBEADM_BOOTSTRAP_FORMAT_IGNITION=true +export AZURE_TENANT_ID="" +export AZURE_CLIENT_ID="" +export AZURE_CLIENT_ID_USER_ASSIGNED_IDENTITY=$AZURE_CLIENT_ID # for compatibility with CAPZ v1.16 templates +export AZURE_CLIENT_SECRET="" +export AZURE_RESOURCE_GROUP="capz-demo" +``` + +From now, you can just copy-paste: +```bash +# Settings needed for AzureClusterIdentity used by the AzureCluster +export AZURE_CLUSTER_IDENTITY_SECRET_NAME="cluster-identity-secret" +export CLUSTER_IDENTITY_NAME="cluster-identity" +export AZURE_CLUSTER_IDENTITY_SECRET_NAMESPACE="default" + +# Create a secret to include the password of the Service Principal identity created in Azure +# This secret will be referenced by the AzureClusterIdentity used by the AzureCluster +kubectl create secret generic "${AZURE_CLUSTER_IDENTITY_SECRET_NAME}" --from-literal=clientSecret="${AZURE_CLIENT_SECRET}" --namespace "${AZURE_CLUSTER_IDENTITY_SECRET_NAMESPACE}" + +# Finally, initialize the management cluster +clusterctl init --infrastructure azure +``` + +Now, you can generate the workload cluster configuration: + +_Notes_: +* at this time, the CAPZ Flatcar sysext PR is still opened (https://github.com/kubernetes-sigs/cluster-api-provider-azure/pull/4575) which means that `--infrastructure azure --flavor flatcar-sysext` must be replaced by `--from /path/to/flatcar-sysext/template.yaml` +* Kubernetes version must match sysext-bakery [releases](https://github.com/flatcar/sysext-bakery/releases/tag/latest) + +```bash +clusterctl generate cluster capi-quickstart \ + --infrastructure azure \ + --kubernetes-version v1.31.1 \ + --control-plane-machine-count=3 \ + --worker-machine-count=3 \ + --flavor flatcar-sysext \ + > "${AZURE_RESOURCE_GROUP}.yaml" +yq -i "with(. | select(.kind == \"AzureClusterIdentity\"); .spec.type |= \"ServicePrincipal\" | .spec.clientSecret.name |= \"${AZURE_CLUSTER_IDENTITY_SECRET_NAME}\" | .spec.clientSecret.namespace |= \"${AZURE_CLUSTER_IDENTITY_SECRET_NAMESPACE}\")" "${AZURE_RESOURCE_GROUP}.yaml" +kubectl apply -f "${AZURE_RESOURCE_GROUP}.yaml" +``` + +After a few minutes, the cluster should be available using latest Flatcar version available on the Azure gallery. + +```bash +clusterctl get kubeconfig "${AZURE_RESOURCE_GROUP}" > "${AZURE_RESOURCE_GROUP}.kubeconfig" +kubectl --kubeconfig "${AZURE_RESOURCE_GROUP}.kubeconfig" get nodes -o wide +``` From 75ba3952f7528354efcc3eee9afbc4780a232ed4 Mon Sep 17 00:00:00 2001 From: Thilo Fromm Date: Tue, 29 Oct 2024 16:56:37 +0100 Subject: [PATCH 2/4] CAPZ-demo: add automation This change adds automation to prepare and to run the CAPZ demo. Automation currently works around limitations caused by the CAPZ upstream PR not being merged. There are TODOs in capz-demo.env for when the PR is merged upstream. Signed-off-by: Thilo Fromm --- CAPZ-sysext/.gitignore | 3 + CAPZ-sysext/azure.env.template | 28 ++ CAPZ-sysext/capz-demo.env | 232 ++++++++++++ .../cluster-template-flatcar-sysext.yaml | 337 ++++++++++++++++++ 4 files changed, 600 insertions(+) create mode 100644 CAPZ-sysext/.gitignore create mode 100644 CAPZ-sysext/azure.env.template create mode 100644 CAPZ-sysext/capz-demo.env create mode 100644 CAPZ-sysext/cluster-template-flatcar-sysext.yaml diff --git a/CAPZ-sysext/.gitignore b/CAPZ-sysext/.gitignore new file mode 100644 index 0000000..4b32c8e --- /dev/null +++ b/CAPZ-sysext/.gitignore @@ -0,0 +1,3 @@ +*.swp +demo +azure.env diff --git a/CAPZ-sysext/azure.env.template b/CAPZ-sysext/azure.env.template new file mode 100644 index 0000000..9822b44 --- /dev/null +++ b/CAPZ-sysext/azure.env.template @@ -0,0 +1,28 @@ +# Template for Azure CAPZ settings + +# The subscrition ID to use for the workload cluster. +export AZURE_SUBSCRIPTION_ID="TODO add subscription ID" + +# From https://capz.sigs.k8s.io/getting-started: +# az ad sp create-for-rbac --role contributor --scopes="/subscriptions/${AZURE_SUBSCRIPTION_ID}" +export AZURE_TENANT_ID="TODO add 'tenant' from output of az command" +export AZURE_CLIENT_ID="TODO add 'appId from output of az command'" +export AZURE_CLIENT_SECRET="TODO add 'password' from output of az command" + +# Uncomment and set this to the base64 encoded public component +# of the SSH key you want to use to log into the control plane node, e.g.: +# base64 -w0 ~/.ssh/id_rsa.pub +# Leave commented out if you don't need SSH access. +#export AZURE_SSH_PUBLIC_KEY_B64="" + + +# +# These usually do not need to be touched +# + +export AZURE_CLIENT_ID_USER_ASSIGNED_IDENTITY="${AZURE_CLIENT_ID}" # for compatibility with CAPZ v1.16 templates + +# AZURE_RESOURCE_GROUP is set in capz-demo.env +export AZURE_CLUSTER_IDENTITY_SECRET_NAME="${AZURE_RESOURCE_GROUP}-cluster-identity-secret" +export CLUSTER_IDENTITY_NAME="${AZURE_RESOURCE_GROUP}-cluster-identity" +export AZURE_CLUSTER_IDENTITY_SECRET_NAMESPACE="default" diff --git a/CAPZ-sysext/capz-demo.env b/CAPZ-sysext/capz-demo.env new file mode 100644 index 0000000..d26d983 --- /dev/null +++ b/CAPZ-sysext/capz-demo.env @@ -0,0 +1,232 @@ +#!/bin/bash + +KUBERNETES_VERSION="v1.30.0" + +AZURE_RESOURCE_GROUP="flatcar-capi-demo-azure" +export AZURE_RESOURCE_GROUP + +WORKER_CONTROLPLANE_NODES=1 +WORKER_NODES=2 + +p() { + echo + echo "#####################################" + echo -e "${@}" + echo "-------------------------------------" +} +# -- + +check_command() { + local cmd="$1" + + if ! command -v "$cmd" &> /dev/null ; then + echo "'$cmd' could not be found. Please install your distro's '$cmd'." + return 1 + fi + echo " - '$cmd'" +} +# -- + +check_file() { + local f="$1" + + if [ ! -f "./$f" ] ; then + echo "prerequisite '$f' could not be found." + return 1 + fi + + echo " - '$f'" +} +# -- + +get_prerequisites() { + p "Prerequisites: Checking for required host commands." + check_command "kubectl" || return + check_command "yq" || return + check_command "wget" || return + + p "Prerequisites: Checking for prerequisite files." + check_file "azure.env" || return + check_file "cluster-template-flatcar-sysext.yaml" || return + + if [ ! -f ./clusterctl ] ; then + p "Prerequisites: fetching clusterctl" + wget https://github.com/kubernetes-sigs/cluster-api/releases/latest/download/clusterctl-linux-amd64 + mv clusterctl-linux-amd64 clusterctl + chmod 755 clusterctl + fi + + if [ ! -f ./kind ] ; then + p "Prerequisites: fetching kind" + wget https://github.com/kubernetes-sigs/kind/releases/latest/download/kind-linux-amd64 + mv kind-linux-amd64 kind + chmod 755 kind + fi + + if [ ! -f ./helm ] ; then + p "Prerequisites: fetching helm" + curl https://get.helm.sh/helm-v3.16.2-linux-amd64.tar.gz \ + | tar xz linux-amd64/helm -O >helm + chmod 755 helm + mkdir -p helm-cache + fi +} +# -- + +setup_kind_cluster() { + p "Bootsstrapping cluster" + ./kind create cluster --kubeconfig=./kind-mgmt.kubeconfig + export EXP_KUBEADM_BOOTSTRAP_FORMAT_IGNITION=true +} +# -- + +kc_mgmt() { + kubectl --kubeconfig=./kind-mgmt.kubeconfig "${@}" + +} +# -- + +kc_worker() { + kubectl --kubeconfig=./${AZURE_RESOURCE_GROUP}.kubeconfig "${@}" +} +# -- + +generate_capz_yaml() { + source ./azure.env + + p "Initialising ClusterAPI Azure provider." + ./clusterctl init --infrastructure azure --kubeconfig=./kind-mgmt.kubeconfig + + # FIXME: add + # --infrastructure azure \ + # --flavor flatcar-sysext \ + # and remove + # --from cluster-template-flatcar-sysext.yaml \ + # when https://github.com/kubernetes-sigs/cluster-api-provider-azure/pull/4575 is merged. + + # TODO: remove when PR is merged + export AZURE_CONTROL_PLANE_MACHINE_TYPE="Standard_D2s_v3" + export AZURE_LOCATION="northeurope" + export AZURE_NODE_MACHINE_TYPE="Standard_D2s_v3" + + export CI_RG="${AZURE_RESOURCE_GROUP}" + export FLATCAR_VERSION="latest" + + p "Generating ${AZURE_RESOURCE_GROUP}.yaml." + ./clusterctl generate cluster ${AZURE_RESOURCE_GROUP} \ + --from cluster-template-flatcar-sysext.yaml \ + --kubeconfig=./kind-mgmt.kubeconfig \ + --kubernetes-version "${KUBERNETES_VERSION}" \ + --control-plane-machine-count=${WORKER_CONTROLPLANE_NODES} \ + --worker-machine-count=${WORKER_NODES} \ + > ${AZURE_RESOURCE_GROUP}.yaml + + yq -i "with(. | select(.kind == \"AzureClusterIdentity\"); .spec.type |= \"ServicePrincipal\" | .spec.clientSecret.name |= \"${AZURE_CLUSTER_IDENTITY_SECRET_NAME}\" | .spec.clientSecret.namespace |= \"${AZURE_CLUSTER_IDENTITY_SECRET_NAMESPACE}\")" "${AZURE_RESOURCE_GROUP}.yaml" +} +# -- + +deploy_capz_cluster() { + + p "Creating client secrets and workload cluster" + + kc_mgmt create secret generic "${AZURE_CLUSTER_IDENTITY_SECRET_NAME}" \ + --from-literal=clientSecret="${AZURE_CLIENT_SECRET}" \ + --namespace "${AZURE_CLUSTER_IDENTITY_SECRET_NAMESPACE}" + + sleep 1 + + kc_mgmt apply -f ./${AZURE_RESOURCE_GROUP}.yaml + + p "Waiting for cluster to be provisioned." + while ! kc_mgmt get cluster | grep "${AZURE_RESOURCE_GROUP}" | grep -i provisioned ; do + sleep 1 + done + + # Hack alert: sometimes ended up with an empty kubeconfig from the command below, so + # we add a defensive sleep. + sleep 2 + + ./clusterctl get kubeconfig ${AZURE_RESOURCE_GROUP} \ + --kubeconfig=./kind-mgmt.kubeconfig \ + --namespace default \ + > ./${AZURE_RESOURCE_GROUP}.kubeconfig + + p "Waiting for all nodes to come up." + local count=0 + local target_count=$((WORKER_CONTROLPLANE_NODES + WORKER_NODES)) + while [ "$count" -lt "$target_count" ] ; do + count="$(kc_worker --request-timeout=5s get nodes \ + | grep "${AZURE_RESOURCE_GROUP}" \ + | wc -l)" + echo "$count of $target_count nodes are up." + done + + local worker_cidr="$(kc_mgmt get cluster "${AZURE_RESOURCE_GROUP}" \ + -o=jsonpath='{.spec.clusterNetwork.pods.cidrBlocks[0]}')" + + p "Deploying Calico to cluster CIDR '$worker_cidr' so worker nodes can talk to each other" + + helm_wrapper() { + ./helm --kubeconfig=./${AZURE_RESOURCE_GROUP}.kubeconfig \ + --registry-config=./${AZURE_RESOURCE_GROUP}.helm-registry.json \ + --repository-cache=./helm-cache \ + --repository-config=./${AZURE_RESOURCE_GROUP}.helm-registry.yaml \ + ${@} + } + + helm_wrapper \ + repo add projectcalico https://docs.tigera.io/calico/charts + + helm_wrapper \ + install calico projectcalico/tigera-operator \ + --version v3.26.1 \ + -f https://raw.githubusercontent.com/kubernetes-sigs/cluster-api-provider-azure/main/templates/addons/calico/values.yaml \ + --set-string "installation.calicoNetwork.ipPools[0].cidr=${worker_cidr}" \ + --namespace tigera-operator \ + --create-namespace + + p "Installing Azure cluster controller" + helm_wrapper \ + install --repo https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/helm/repo \ + cloud-provider-azure \ + --generate-name --set infra.clusterName=${AZURE_RESOURCE_GROUP} \ + --set "cloudControllerManager.clusterCIDR=${worker_cidr}" \ + --set-string "cloudControllerManager.caCertDir=/usr/share/ca-certificates" + + p "Your nodes should be ready soon" + kc_worker get nodes + + local bastion="$(kc_mgmt get azurecluster flatcar-capi-demo-azure -o json \ + | jq '.spec.networkSpec.apiServerLB.frontendIPs[0].publicIP.dnsName')" + + local ssh="ssh capi@${bastion}" + p "Cluster is deployed and can now be used ('kc_worker' kubectl wrapper)." + p "You can use 'az serial-console connect -g ${AZURE_RESOURCE_GROUP} -n ' connects via serial console, or\n" \ + "'${ssh}' to ssh into the control plane node (if you set AZURE_SSH_PUBLIC_KEY_B64 in azure.env)." +} +# -- + +cleanup() { + kc_mgmt delete cluster ${AZURE_RESOURCE_GROUP} + ./kind delete cluster --kubeconfig=./kind-mgmt.kubeconfig +} +# -- + +help() { + cat < /tmp/kubernetes" + ExecStartPre=/usr/lib/systemd/systemd-sysupdate -C kubernetes update + ExecStartPost=/usr/bin/sh -c "readlink --canonicalize /etc/extensions/kubernetes.raw > /tmp/kubernetes-new" + ExecStartPost=/usr/bin/sh -c "if ! cmp --silent /tmp/kubernetes /tmp/kubernetes-new; then touch /run/reboot-required; fi" + - name: update-engine.service + # Set this to 'false' if you want to enable Flatcar auto-update + mask: ${FLATCAR_DISABLE_AUTO_UPDATE:=true} + - name: locksmithd.service + # NOTE: To coordinate the node reboot in this context, we recommend to use Kured. + mask: true + - name: systemd-sysupdate.timer + # Set this to 'true' if you want to enable the Kubernetes auto-update. + # NOTE: Only patches version will be pulled. + enabled: true + dropins: + - name: bootcheck.conf + contents: | + [Timer] + OnBootSec=1min + OnUnitActiveSec=10s + RandomizedDelaySec=1s + - name: kubeadm.service + dropins: + - name: 10-flatcar.conf + contents: | + [Unit] + After=oem-cloudinit.service + # kubeadm must run after containerd - see https://github.com/kubernetes-sigs/image-builder/issues/939. + After=containerd.service + joinConfiguration: + nodeRegistration: + kubeletExtraArgs: + cloud-provider: external + name: '@@HOSTNAME@@' + postKubeadmCommands: [] + preKubeadmCommands: + - sed -i "s/@@HOSTNAME@@/$(curl -s -H Metadata:true --noproxy '*' 'http://169.254.169.254/metadata/instance?api-version=2020-09-01' + | jq -r .compute.name)/g" /etc/kubeadm.yml +--- +apiVersion: cluster.x-k8s.io/v1beta1 +kind: Cluster +metadata: + name: ${CLUSTER_NAME} + namespace: default +spec: + clusterNetwork: + pods: + cidrBlocks: + - 192.168.0.0/16 + controlPlaneRef: + apiVersion: controlplane.cluster.x-k8s.io/v1beta1 + kind: KubeadmControlPlane + name: ${CLUSTER_NAME}-control-plane + infrastructureRef: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: AzureCluster + name: ${CLUSTER_NAME} +--- +apiVersion: cluster.x-k8s.io/v1beta1 +kind: MachineDeployment +metadata: + name: ${CLUSTER_NAME}-md-0 + namespace: default +spec: + clusterName: ${CLUSTER_NAME} + replicas: ${WORKER_MACHINE_COUNT} + selector: + matchLabels: null + template: + spec: + bootstrap: + configRef: + apiVersion: bootstrap.cluster.x-k8s.io/v1beta1 + kind: KubeadmConfigTemplate + name: ${CLUSTER_NAME}-md-0 + clusterName: ${CLUSTER_NAME} + infrastructureRef: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: AzureMachineTemplate + name: ${CLUSTER_NAME}-md-0 + version: ${KUBERNETES_VERSION} +--- +apiVersion: controlplane.cluster.x-k8s.io/v1beta1 +kind: KubeadmControlPlane +metadata: + name: ${CLUSTER_NAME}-control-plane + namespace: default +spec: + kubeadmConfigSpec: + clusterConfiguration: + apiServer: + extraArgs: + cloud-provider: external + timeoutForControlPlane: 20m + controllerManager: + extraArgs: + allocate-node-cidrs: "false" + cloud-provider: external + cluster-name: ${CLUSTER_NAME} + etcd: + local: + dataDir: /var/lib/etcddisk/etcd + extraArgs: + quota-backend-bytes: "8589934592" + diskSetup: + filesystems: + - device: /dev/disk/azure/scsi1/lun0 + extraOpts: + - -E + - lazy_itable_init=1,lazy_journal_init=1 + filesystem: ext4 + label: etcd_disk + overwrite: false + partitions: [] + files: + - contentFrom: + secret: + key: control-plane-azure.json + name: ${CLUSTER_NAME}-control-plane-azure-json + owner: root:root + path: /etc/kubernetes/azure.json + permissions: "0644" + format: ignition + ignition: + containerLinuxConfig: + additionalConfig: | + systemd: + units: + - name: systemd-sysupdate.service + dropins: + - name: kubernetes.conf + contents: | + [Service] + ExecStartPre=/usr/bin/sh -c "readlink --canonicalize /etc/extensions/kubernetes.raw > /tmp/kubernetes" + ExecStartPre=/usr/lib/systemd/systemd-sysupdate -C kubernetes update + ExecStartPost=/usr/bin/sh -c "readlink --canonicalize /etc/extensions/kubernetes.raw > /tmp/kubernetes-new" + ExecStartPost=/usr/bin/sh -c "if ! cmp --silent /tmp/kubernetes /tmp/kubernetes-new; then touch /run/reboot-required; fi" + - name: update-engine.service + # Set this to 'false' if you want to enable Flatcar auto-update + mask: ${FLATCAR_DISABLE_AUTO_UPDATE:=true} + - name: locksmithd.service + # NOTE: To coordinate the node reboot in this context, we recommend to use Kured. + mask: true + - name: systemd-sysupdate.timer + # Set this to 'true' if you want to enable the Kubernetes auto-update. + # NOTE: Only patches version will be pulled. + enabled: true + dropins: + - name: bootcheck.conf + contents: | + [Timer] + OnBootSec=1min + OnUnitActiveSec=10s + RandomizedDelaySec=1s + - name: kubeadm.service + dropins: + - name: 10-flatcar.conf + contents: | + [Unit] + After=oem-cloudinit.service + # kubeadm must run after containerd - see https://github.com/kubernetes-sigs/image-builder/issues/939. + After=containerd.service + # Workaround for https://github.com/kubernetes-sigs/cluster-api/issues/7679. + storage: + disks: + - device: /dev/disk/azure/scsi1/lun0 + partitions: + - number: 1 + links: + - path: /etc/extensions/kubernetes.raw + hard: false + target: /opt/extensions/kubernetes/kubernetes-${KUBERNETES_VERSION}-x86-64.raw + files: + - path: /etc/sysupdate.kubernetes.d/kubernetes-${KUBERNETES_VERSION%.*}.conf + mode: 0644 + contents: + remote: + url: https://github.com/flatcar/sysext-bakery/releases/download/latest/kubernetes-${KUBERNETES_VERSION%.*}.conf + - path: /etc/sysupdate.d/noop.conf + mode: 0644 + contents: + remote: + url: https://github.com/flatcar/sysext-bakery/releases/download/latest/noop.conf + - path: /opt/extensions/kubernetes/kubernetes-${KUBERNETES_VERSION}-x86-64.raw + contents: + remote: + url: https://github.com/flatcar/sysext-bakery/releases/download/latest/kubernetes-${KUBERNETES_VERSION}-x86-64.raw + initConfiguration: + nodeRegistration: + kubeletExtraArgs: + cloud-provider: external + name: '@@HOSTNAME@@' + joinConfiguration: + nodeRegistration: + kubeletExtraArgs: + cloud-provider: external + name: '@@HOSTNAME@@' + mounts: + - - etcd_disk + - /var/lib/etcddisk + postKubeadmCommands: [] + preKubeadmCommands: + - sed -i "s/@@HOSTNAME@@/$(curl -s -H Metadata:true --noproxy '*' 'http://169.254.169.254/metadata/instance?api-version=2020-09-01' + | jq -r .compute.name)/g" /etc/kubeadm.yml + machineTemplate: + infrastructureRef: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: AzureMachineTemplate + name: ${CLUSTER_NAME}-control-plane + replicas: ${CONTROL_PLANE_MACHINE_COUNT:=1} + version: ${KUBERNETES_VERSION} +--- +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: AzureCluster +metadata: + name: ${CLUSTER_NAME} + namespace: default +spec: + identityRef: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: AzureClusterIdentity + name: ${CLUSTER_IDENTITY_NAME} + location: ${AZURE_LOCATION} + networkSpec: + subnets: + - name: control-plane-subnet + role: control-plane + - name: node-subnet + role: node + vnet: + name: ${AZURE_VNET_NAME:=${CLUSTER_NAME}-vnet} + resourceGroup: ${AZURE_RESOURCE_GROUP:=${CLUSTER_NAME}} + subscriptionID: ${AZURE_SUBSCRIPTION_ID} +--- +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: AzureClusterIdentity +metadata: + labels: + clusterctl.cluster.x-k8s.io/move-hierarchy: "true" + name: ${CLUSTER_IDENTITY_NAME} + namespace: default +spec: + allowedNamespaces: {} + clientID: ${AZURE_CLIENT_ID_USER_ASSIGNED_IDENTITY} + tenantID: ${AZURE_TENANT_ID} + type: ${CLUSTER_IDENTITY_TYPE:=WorkloadIdentity} +--- +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: AzureMachineTemplate +metadata: + name: ${CLUSTER_NAME}-control-plane + namespace: default +spec: + template: + spec: + dataDisks: + - diskSizeGB: 256 + lun: 0 + nameSuffix: etcddisk + image: + marketplace: + version: ${FLATCAR_VERSION} + publisher: kinvolk + offer: flatcar-container-linux-corevm-amd64 + sku: stable-gen2 + osDisk: + diskSizeGB: 128 + osType: Linux + sshPublicKey: ${AZURE_SSH_PUBLIC_KEY_B64:=""} + vmSize: ${AZURE_CONTROL_PLANE_MACHINE_TYPE} +--- +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: AzureMachineTemplate +metadata: + name: ${CLUSTER_NAME}-md-0 + namespace: default +spec: + template: + spec: + image: + marketplace: + version: ${FLATCAR_VERSION} + publisher: kinvolk + offer: flatcar-container-linux-corevm-amd64 + sku: stable-gen2 + osDisk: + diskSizeGB: 128 + osType: Linux + sshPublicKey: ${AZURE_SSH_PUBLIC_KEY_B64:=""} + vmSize: ${AZURE_NODE_MACHINE_TYPE} From 9fd07dc787245105475915179f05e11f572845f9 Mon Sep 17 00:00:00 2001 From: Thilo Fromm Date: Mon, 4 Nov 2024 15:06:12 +0100 Subject: [PATCH 3/4] README: add documentation for demo scripting / automation --- CAPZ-sysext/README.md | 144 ++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 140 insertions(+), 4 deletions(-) diff --git a/CAPZ-sysext/README.md b/CAPZ-sysext/README.md index d5a7eed..366107d 100644 --- a/CAPZ-sysext/README.md +++ b/CAPZ-sysext/README.md @@ -1,10 +1,144 @@ # Cluster API Azure (CAPZ) with Flatcar -This demo is divided into two sections: -* [Cluster API Azure using Flatcar sysext template](#cluster-api-azure-using-flatcar-sysext-template) -* Cluster API Azure using AKS (mixing Ubuntu and Flatcar nodes) -## Cluster API Azure using Flatcar sysext template +This demo is divided into three sections: +* Automated, scripted set-up of a ClusterAPI worker cluster on Azure, including live in-place updates +* Full manual walk-through of setting up [Cluster API Azure using Flatcar sysext template](#manual-cluster-api-azure-using-flatcar-sysext-template) +* Full manual walk-through of setting up Cluster API Azure using AKS (mixing Ubuntu and Flatcar nodes) + +## Automated set-up of a Cluster API demo cluster on Azure + +The automation will lead you through all steps necessary for creating an Azure Cluster API cluster from scratch. +It will also take care of creating a local management cluster, and generally requires very few prerequisites. +The automation is best suitable if you want to start from scratch, and/or if you need a quick throw-away CAPZ cluster. + +**tl;dr** + +```bash +git clone https://github.com/flatcar/flatcar-demos.git +cd flatcar-demos/CAPZ-sysext +mkdir demo +cd demo + +cp ../cluster-template-flatcar-sysext.yaml . +cp ../azure.env.template azure.env +vim azure.env # fill in account information + +bash -i +source ../capz-demo.env + +get_prerequisites +setup_kind_cluster +generate_capz_yaml + +deploy_capz_cluster + +kc_worker get nodes -o wide +``` + +After worker nodes are operational +```bash +kc_worker apply -f kured-dockerhub.yaml +``` + +Watch the in-place update +```bash +watch 'source ../capz-demo.env; kc_worker get nodes -o wide;' +``` + +### Prerequisites + +An Azure account and an active subscription. +We will create an RBAC account for use by the Cluster API automation, so you'll need access to the Active Directory of your account. + +### Prepare the demo + +1. Clone the flatcar-demos repo and change into the "CAPZ-sysext" subdirectory of the repo. + ```bash + git clone https://github.com/flatcar/flatcar-demos.git + cd flatcar-demos/CAPZ-sysext + ``` +2. Create a sub-directory `demo` to work in (will be ignored as per the repo's `.gitgnore`). + ```bash + mkdir demo + cd demo + ``` +3. Copy the upper directory's `cluster-template-flatcar-sysext.yaml`, and copy `azure.env.template` to `azure.env`: + ```bash + cp ../cluster-template-flatcar-sysext.yaml . + cp ../azure.env.template azure.env + ``` +4. Edit `azure.env` and fill in the required variables. + Comments in the file will guide you through creating an Active Directory RBAC account to use for the demo if you don't have one available. + You can also optionally configure an SSH key to log in to the worker cluster's control plane node. + +You should now have: +- the automation available locally from the flatcar-demos repository you've cloned +- a `demo` sub-directory with two files in it: + - `cluster-template-flatcar-sysext.yaml` which the automation will use to generate the worker cluster configuration from + - `azure.env` with data to access the Azure account you want to use for the demo + +### Run the demo + +The demo will first create a local KIND cluster and install the Cluster API Azure provider. +It will then generate a cluster configuration for the workload cluster on Azure, and apply it. +It will wait for the Azure cluster to come up, then give control back to you. + +You can interact with the KIND management cluster by using `kc_mgmt `, and with the workload cluster via `kc_worker `. +`kc_mgmt` and `kc_worker` are just wrappers around kubectl with the respective `kubeconfig` set. + +1. Start a new shell and source the automation + ```bash + bash -i + source ../capz-demo.env + ``` +2. Check prerequisites. This will download `kind` and `helm` for local use. + ```bash + get_prerequisites + ``` +3. Set up the KIND (Kubernetes-IN-Docker") cluster for local use. + ```bash + setup_kind_cluster + ``` +4. Install the Azure provider and generate the Cluster API Azure cluster configuration. + ```bash + generate_capz_yaml + ``` +5. Provision the ClusterAPI Azure workload cluster. + This will apply the cluster configuration to the management cluster, which will start the provisioning process on Azure. + It will then wait until the cluster is fully provisioned, and install cluster add-ons (Calico and the external cloud provider) + to make the cluster operational. + Provisioning the cluster can take some time. + You can watch the resources being created in the `flatcar-capi-demo-azure` resource group used by the automation in the Azure portal. + ```bash + deploy_capz_cluster + ``` + +You can now interact with the workload cluster using `kc_worker`, a simple wrapper around `kubectl`. + +#### Live in-place Kubernetes updates + +The `cluster-template-flatcar-sysext.yaml` shipped with this repo has in-place Kubernetes updates enabled via `systemd-sysupdate`. +An update should already have been staged on the workload cluster (happens almost immediately after provisioning). +The `systemd-sysupdate` configuration will also have created a flag file on the node to signal a need for reboot: `/run/reboot-required`. + +To demo live in-place Kubernetes updates, all we need to do is to provision KureD to the workload cluster. +A suitable kured configuration is shipped with the repository. + +Run +```bash +kc_worker apply -f ../kured-dockerhub.yaml +``` +to start the process. + +KureD will detect the flag file and evacuate and reboot the nodes, one after another. +You can watch the process: +```bash +watch 'source ../capz-demo.env; kc_worker get nodes;' +``` + + +## Manual Cluster API Azure using Flatcar sysext template In this demo, you will learn how to create a Kubernetes cluster using Azure resources and powered by Flatcar nodes using the systemd-sysext approach. This is inspired from: https://capz.sigs.k8s.io/getting-started @@ -16,6 +150,8 @@ In this demo, you will learn how to create a Kubernetes cluster using Azure reso * A management cluster (e.g any existing Kubernetes cluster) * `clusterctl` and `yq` up-to-date and available in the `$PATH` +Contrary to the automated set-up, which creates a local KIND cluster for management, the below assumes you already run a management cluster. + ### Initialize the management cluster We first need to export some variables and create some secrets before initializing the management cluster: From 94f91f3801220aea66d1cc3a27312d01ccb31abd Mon Sep 17 00:00:00 2001 From: Mathieu Tortuyaux Date: Mon, 4 Nov 2024 15:15:35 +0100 Subject: [PATCH 4/4] readme: mention CCM and CNI Signed-off-by: Mathieu Tortuyaux Co-authored-by: Thilo Fromm --- CAPZ-sysext/README.md | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/CAPZ-sysext/README.md b/CAPZ-sysext/README.md index 366107d..18205f4 100644 --- a/CAPZ-sysext/README.md +++ b/CAPZ-sysext/README.md @@ -4,7 +4,7 @@ This demo is divided into three sections: * Automated, scripted set-up of a ClusterAPI worker cluster on Azure, including live in-place updates * Full manual walk-through of setting up [Cluster API Azure using Flatcar sysext template](#manual-cluster-api-azure-using-flatcar-sysext-template) -* Full manual walk-through of setting up Cluster API Azure using AKS (mixing Ubuntu and Flatcar nodes) +* Full manual walk-through of setting up Cluster API Azure using AKS (mixing Ubuntu and Flatcar nodes) (TBD) ## Automated set-up of a Cluster API demo cluster on Azure @@ -163,6 +163,7 @@ export AZURE_CLIENT_ID="" export AZURE_CLIENT_ID_USER_ASSIGNED_IDENTITY=$AZURE_CLIENT_ID # for compatibility with CAPZ v1.16 templates export AZURE_CLIENT_SECRET="" export AZURE_RESOURCE_GROUP="capz-demo" +export AZURE_LOCATION="centralus" ``` From now, you can just copy-paste: @@ -180,6 +181,8 @@ kubectl create secret generic "${AZURE_CLUSTER_IDENTITY_SECRET_NAME}" --from-lit clusterctl init --infrastructure azure ``` +### Create the workload cluster + Now, you can generate the workload cluster configuration: _Notes_: @@ -187,7 +190,7 @@ _Notes_: * Kubernetes version must match sysext-bakery [releases](https://github.com/flatcar/sysext-bakery/releases/tag/latest) ```bash -clusterctl generate cluster capi-quickstart \ +clusterctl generate cluster ${AZURE_RESOURCE_GROUP} \ --infrastructure azure \ --kubernetes-version v1.31.1 \ --control-plane-machine-count=3 \ @@ -204,3 +207,13 @@ After a few minutes, the cluster should be available using latest Flatcar versio clusterctl get kubeconfig "${AZURE_RESOURCE_GROUP}" > "${AZURE_RESOURCE_GROUP}.kubeconfig" kubectl --kubeconfig "${AZURE_RESOURCE_GROUP}.kubeconfig" get nodes -o wide ``` + +Of course, the nodes will not be ready while CNI and CCM are not deployed, here's a simple example using Calico: +``` +# CNI +export IPV4_CIDR_BLOCK=$(kubectl get cluster "${AZURE_RESOURCE_GROUP}" -o=jsonpath='{.spec.clusterNetwork.pods.cidrBlocks[0]}') +helm repo add projectcalico https://docs.tigera.io/calico/charts && \ +helm install calico projectcalico/tigera-operator --version v3.26.1 -f https://raw.githubusercontent.com/kubernetes-sigs/cluster-api-provider-azure/main/templates/addons/calico/values.yaml --set-string "installation.calicoNetwork.ipPools[0].cidr=${IPV4_CIDR_BLOCK}" --namespace tigera-operator --create-namespace +# CCM +helm install --repo https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/helm/repo cloud-provider-azure --generate-name --set infra.clusterName=${AZURE_RESOURCE_GROUP} --set "cloudControllerManager.clusterCIDR=${IPV4_CIDR_BLOCK}" --set-string "cloudControllerManager.caCertDir=/usr/share/ca-certificates" +```