kubernetes: add sysext documentation

Signed-off-by: Mathieu Tortuyaux <[email protected]>

docs/container-runtimes/getting-started-with-kubernetes.md

@@ -40,15 +40,72 @@ _Known issues_:
 
 ## Using Kubeadm
 
-`kubeadm` remains one standard way to quickly deploy and operate a Kubernetes cluster. It's possible to install the tools (`kubeadm`, `kubelet`, etc.) using Ignition.
+`kubeadm` remains one standard way to quickly deploy and operate a Kubernetes cluster. It's possible to install the tools (`kubeadm`, `kubelet`, etc.) using Ignition or directly with the Kubernetes sysext image distributed from the [flatcar/sysext-bakery][sysext-bakery] release page.
 
 ### Setup the control plane
 
-Here's an example with [butane][butane] to setup a control plane.
-
-:warning: To ease the reading, we voluntarily omitted the checksums of the downloaded artifacts.
-
-```yaml
+Here's an example with [butane][butane] to setup a control plane using the Systemd sysext approach or the binaries approach.
+
+<div>
+  <ul class="nav nav-tabs">
+    <li class="active"><a href="#sysext" data-toggle="tab">With Systemd sysext</a></li>
+    <li><a href="#no-sysext" data-toggle="tab">Without Systemd sysext</a></li>
+  </ul>
+  <div class="tab-content coreos-docs-image-table">
+    <div class="tab-pane" id="sysext">
+      <div class="channel-info">
+        This is an example using Systemd sysext and Systemd sysupdate:
+        <pre>
+---
+version: 1.0.0
+variant: flatcar
+storage:
+  links:
+    - target: /opt/extensions/kubernetes/kubernetes-v1.27.4-x86-64.raw
+      path: /etc/extensions/kubernetes.raw
+      hard: false
+  files:
+    - path: /etc/sysupdate.kubernetes.d/kubernetes.conf
+      contents:
+        source: https://github.com/flatcar/sysext-bakery/releases/download/20230901/kubernetes.conf
+    - path: /etc/sysupdate.d/noop.conf
+      contents:
+        source: https://github.com/flatcar/sysext-bakery/releases/download/20230901/noop.conf
+    - path: /opt/extensions/kubernetes/kubernetes-v1.27.4-x86-64.raw
+      contents:
+        source: https://github.com/flatcar/sysext-bakery/releases/download/20230901/kubernetes-v1.27.4-x86-64.raw
+systemd:
+  units:
+    - name: systemd-sysupdate.timer
+      enabled: true
+    - name: systemd-sysupdate.service
+      dropins:
+        - name: kubernetes.conf
+          contents: |
+            [Service]
+            ExecStartPre=/usr/lib/systemd/systemd-sysupdate -C kubernetes update
+    - name: kubeadm.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=Kubeadm service
+        Requires=containerd.service
+        After=containerd.service
+        ConditionPathExists=!/etc/kubernetes/kubelet.conf
+        [Service]
+        ExecStartPre=/usr/bin/kubeadm init
+        ExecStartPre=/usr/bin/mkdir /home/core/.kube
+        ExecStartPre=/usr/bin/cp /etc/kubernetes/admin.conf /home/core/.kube/config
+        ExecStart=/usr/bin/chown -R core:core /home/core/.kube
+        [Install]
+        WantedBy=multi-user.target
+        </pre>
+      </div>
+    </div>
+    <div class="tab-pane" id="no-sysext">
+      <div class="channel-info">
+        :warning: To ease the reading, we voluntarily omitted the checksums of the downloaded artifacts.
+        <pre>
 ---
 version: 1.0.0
 variant: flatcar
@@ -104,18 +161,21 @@ systemd:
         Requires=containerd.service
         After=containerd.service
         ConditionPathExists=!/etc/kubernetes/kubelet.conf
-
         [Service]
         Environment="PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/bin"
         ExecStartPre=/opt/bin/kubeadm config images pull
         ExecStartPre=/opt/bin/kubeadm init --config /etc/kubeadm.yml
         ExecStartPre=/usr/bin/mkdir /home/core/.kube
         ExecStartPre=/usr/bin/cp /etc/kubernetes/admin.conf /home/core/.kube/config
         ExecStart=/usr/bin/chown -R core:core /home/core/.kube
-
         [Install]
         WantedBy=multi-user.target
-```
+        </pre>
+      </div>
+    </div>
+  </div>
+</div>
+
 
 This minimal configuration can be used with Flatcar on QEMU (:warning: be sure that the instance has enough memory: 4096mb is good).
 
@@ -142,7 +202,62 @@ We can now prepare the nodes to join the cluster.
 
 Here's the [butane][butane] configuration to setup the nodes.
 
-```yaml
+<div>
+  <ul class="nav nav-tabs">
+    <li class="active"><a href="#sysext" data-toggle="tab">With Systemd sysext</a></li>
+    <li><a href="#no-sysext" data-toggle="tab">Without Systemd sysext</a></li>
+  </ul>
+  <div class="tab-content coreos-docs-image-table">
+    <div class="tab-pane" id="sysext">
+      <div class="channel-info">
+        This is an example using Systemd sysext and Systemd sysupdate:
+        <pre>
+---
+version: 1.0.0
+variant: flatcar
+storage:
+  links:
+    - target: /opt/extensions/kubernetes/kubernetes-v1.27.4-x86-64.raw
+      path: /etc/extensions/kubernetes.raw
+      hard: false
+  files:
+    - path: /etc/sysupdate.kubernetes.d/kubernetes.conf
+      contents:
+        source: https://github.com/flatcar/sysext-bakery/releases/download/20230901/kubernetes.conf
+    - path: /etc/sysupdate.d/noop.conf
+      contents:
+        source: https://github.com/flatcar/sysext-bakery/releases/download/20230901/noop.conf
+    - path: /opt/extensions/kubernetes/kubernetes-v1.27.4-x86-64.raw
+      contents:
+        source: https://github.com/flatcar/sysext-bakery/releases/download/20230901/kubernetes-v1.27.4-x86-64.raw
+systemd:
+  units:
+    - name: systemd-sysupdate.timer
+      enabled: true
+    - name: systemd-sysupdate.service
+      dropins:
+        - name: kubernetes.conf
+          contents: |
+            [Service]
+            ExecStartPre=/usr/lib/systemd/systemd-sysupdate -C kubernetes update
+    - name: kubeadm.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=Kubeadm service
+        Requires=containerd.service
+        After=containerd.service
+        [Service]
+        ExecStart=/usr/bin/kubeadm join $(output from 'kubeadm token create --print-join-command')
+        [Install]
+        WantedBy=multi-user.target
+        </pre>
+      </div>
+    </div>
+    <div class="tab-pane" id="no-sysext">
+      <div class="channel-info">
+        :warning: To ease the reading, we voluntarily omitted the checksums of the downloaded artifacts.
+        <pre>
 ---
 version: 1.0.0
 variant: flatcar
@@ -179,14 +294,16 @@ systemd:
         Description=Kubeadm service
         Requires=containerd.service
         After=containerd.service
-
         [Service]
         Environment="PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/bin"
-        ExecStart=/opt/bin/kubeadm join <output from 'kubeadm token create --print-join-command'>
-
+        ExecStart=/opt/bin/kubeadm join $(output from 'kubeadm token create --print-join-command')
         [Install]
         WantedBy=multi-user.target
-```
+        </pre>
+      </div>
+    </div>
+  </div>
+</div>
 
 This method is far from being ideal in terms of infrastructure as code as it requires a two steps manipulation: create the control plane to generate the join configuration then pass that configuration to the nodes. Other solutions exist to make things easier, like Cluster API or [Typhoon][typhoon].
 
@@ -244,4 +361,5 @@ Based on users feedback, Flatcar is known to work with Kubespray - you can read
 [kubespray-documentation]: https://kubespray.io
 [kubespray-documentation-flatcar]: https://kubespray.io/#/docs/flatcar
 [openstack]: https://cluster-api-openstack.sigs.k8s.io/clusteropenstack/configuration.html#ignition-based-images
+[sysext-bakery]: https://github.com/flatcar/sysext-bakery
 [typhoon]: https://typhoon.psdn.io/