Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bake k0s as a systemd sysext image. #99

Closed
wants to merge 3 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
109 changes: 109 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,7 @@ For extensions that are not part of the GitHub Release or which you want to cust
| `wasmcloud` | released |
| `tailscale` | released |
| `crio` | released |
| `k0s` | released |
| `k3s` | released |
| `rke2` | released |
| `keepalived` | build script |
Expand Down Expand Up @@ -223,6 +224,114 @@ storage:
Of course its also possible to use the
[artifact-follower](https://falco.org/blog/falcoctl-install-manage-rules-plugins/#follow-artifacts) to download falco artifacts automatically.

#### k0s

To setup [k0s](https://docs.k0sproject.io/stable/) we need:

1. the sysext plus the configuration files /etc/k0s/k0s.yaml and/or /etc/default/k0s
2. the systemd units

After you enable k0s syext following the guide deacribed on Consuming the published images, follow the next steps:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
After you enable k0s syext following the guide deacribed on Consuming the published images, follow the next steps:
After you enabled k0s sysext following the guide described on consuming the published images, follow the next steps:


1.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this mandatory? One might not know by advance the private IP address.

a) Customize /etc/k0s/k0s.yaml for either controller/worker ([More info](https://docs.k0sproject.io/stable/configuration/#configuration-file-reference)). Below is the default configuration for k0s implemented via Ignition which you should change it according to your needs:

```yaml
storage:
files:
- path: /etc/k0s/k0s.yaml
overwrite: true
mode: 0644
contents:
inline: |
apiVersion: k0s.k0sproject.io/v1beta1
kind: ClusterConfig
metadata:
name: k0s
spec:
api:
address: 192.168.68.104
k0sApiPort: 9443
port: 6443
sans:
- 192.168.68.104
controllerManager: {}
extensions:
helm:
concurrencyLevel: 5
installConfig:
users:
etcdUser: etcd
kineUser: kube-apiserver
konnectivityUser: konnectivity-server
kubeAPIserverUser: kube-apiserver
kubeSchedulerUser: kube-scheduler
konnectivity:
adminPort: 8133
agentPort: 8132
network:
clusterDomain: cluster.local
dualStack:
enabled: false
kubeProxy:
iptables:
minSyncPeriod: 0s
syncPeriod: 0s
ipvs:
minSyncPeriod: 0s
syncPeriod: 0s
tcpFinTimeout: 0s
tcpTimeout: 0s
udpTimeout: 0s
metricsBindAddress: 0.0.0.0:10249
mode: iptables
kuberouter:
autoMTU: true
hairpin: Enabled
metricsPort: 8080
nodeLocalLoadBalancing:
enabled: false
envoyProxy:
apiServerBindPort: 7443
konnectivityServerBindPort: 7132
type: EnvoyProxy
podCIDR: 10.244.0.0/16
provider: kuberouter
serviceCIDR: 10.96.0.0/12
scheduler: {}
storage:
etcd:
peerAddress: 192.168.68.104
type: etcd
telemetry:
enabled: true
```

b) If you want use another CRI other than k0s embedded one define it here: /etc/default/k0s

```yaml
storage:
files:
- path: /etc/default/k0s
overwrite: true
mode: 0644
contents:
inline: |
CRI_SOCKET="remote:unix:///var/run/custom_CRI.sock"
```

2. Defined systemd units are: k0s.service, k0scontroller.service and k0sworker.service
a) If you go with the embedded k0s components you have to enable k0scontroller.service or k0sworker.service according to the `ROLE`(controller/worker) of the node and the /etc/k0s/k0s.yaml you have defined at step 1

```yaml
systemd:
units:
- name: k0s`ROLE`.service
enabled: true
```

b) If you decide to go with custom components for k0s (different CRI, external ETCD cluster, ...) you have to enable k0s.service for controller or k0sworker.service for worker and, of course, configure /etc/k0s/k0s.yaml accordingly.

#### Kubernetes

The [Flatcar Kubernetes docs](https://www.flatcar.org/docs/latest/container-runtimes/getting-started-with-kubernetes/) show how to use the extension provided here for controllers and workers.
Expand Down
130 changes: 130 additions & 0 deletions create_k0s_sysext.sh
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file needs to be pushed with executable mode:

diff --git a/create_k0s_sysext.sh b/create_k0s_sysext.sh
old mode 100644
new mode 100755

Original file line number Diff line number Diff line change
@@ -0,0 +1,130 @@
#!/usr/bin/env bash
set -euo pipefail

export ARCH="${ARCH-x86-64}"
SCRIPTFOLDER="$(dirname "$(readlink -f "$0")")"

if [ $# -lt 2 ] || [ "$1" = "-h" ] || [ "$1" = "--help" ]; then
echo "Usage: $0 VERSION SYSEXTNAME"
echo "The script will download the k0s binary (e.g., for v1.31.2+k0s.0) and create a sysext squashfs image with the name SYSEXTNAME.raw in the current folder."
echo "A temporary directory named SYSEXTNAME in the current folder will be created and deleted again."
echo "All files in the sysext image will be owned by root."
echo "To use arm64 pass 'ARCH=arm64' as environment variable (current value is '${ARCH}')."
"${SCRIPTFOLDER}"/bake.sh --help
exit 1
fi

VERSION="$1"
SYSEXTNAME="$2"

# The github release uses different arch identifiers, we map them here
# and rely on bake.sh to map them back to what systemd expects
if [ "${ARCH}" = "amd64" ] || [ "${ARCH}" = "x86-64" ]; then
ARCH="amd64"
fi
if [ "${ARCH}" = "arm64" ] || [ "${ARCH}" = "aarch64" ]; then
ARCH="arm64"
fi

URL="https://github.com/k0sproject/k0s/releases/download/${VERSION}/k0s-${VERSION}-${ARCH}"

rm -rf "${SYSEXTNAME}"
mkdir -p "${SYSEXTNAME}"/usr/local/bin
curl -o "${SYSEXTNAME}/usr/local/bin/k0s" -fsSL "${URL}"
chmod +x "${SYSEXTNAME}"/usr/local/bin/k0s
pushd "${SYSEXTNAME}"/usr/local/bin/
ln -s ./k0s kubectl
ln -s ./k0s ctr
popd

mkdir -p "${SYSEXTNAME}"/usr/local/lib/systemd/system/
cat > "${SYSEXTNAME}"/usr/local/lib/systemd/system/k0s.service << EOF
[Unit]
Description=k0s - Init Controller / External ETCD Controller
Documentation=https://docs.k0sproject.io
ConditionFileIsExecutable=/usr/local/bin/k0s

Requires=containerd.service
Wants=network-online.target
After=network-online.target containerd.service

[Service]
EnvironmentFile=-/etc/default/k0s
StartLimitInterval=5
StartLimitBurst=10
ExecStart=/bin/sh -c '[ -n "${CRI_SOCKET}" ] && exec /usr/local/bin/k0s controller --config=/etc/k0s/k0s.yaml --cri-socket=${CRI_SOCKET} || exec /usr/local/bin/k0s controller --config=/etc/k0s/k0s.yaml'
tormath1 marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This won't work:

./create_k0s_sysext.sh: line 41: CRI_SOCKET: unbound variable

You need to escape those values (individually or globally with \EOF)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
ExecStart=/bin/sh -c '[ -n "${CRI_SOCKET}" ] && exec /usr/local/bin/k0s controller --config=/etc/k0s/k0s.yaml --cri-socket=${CRI_SOCKET} || exec /usr/local/bin/k0s controller --config=/etc/k0s/k0s.yaml'
ExecStart=/bin/sh -c '[ -n "${CRI_SOCKET}" ] && exec /usr/local/bin/k0s controller --config=/etc/k0s/k0s.yaml --cri-socket="${CRI_SOCKET}" || exec /usr/local/bin/k0s controller --config=/etc/k0s/k0s.yaml'

(Same for the other units)


RestartSec=10
Delegate=yes
KillMode=process
LimitCORE=infinity
TasksMax=infinity
TimeoutStartSec=0
LimitNOFILE=999999
Restart=always

[Install]
WantedBy=multi-user.target
EOF

cat > "${SYSEXTNAME}"/usr/local/lib/systemd/system/k0scontroller.service << EOF
[Unit]
Description=k0s - Controller
Documentation=https://docs.k0sproject.io
ConditionFileIsExecutable=/usr/local/bin/k0s

Requires=containerd.service
Wants=network-online.target
After=network-online.target containerd.service

[Service]
EnvironmentFile=-/etc/default/k0s
StartLimitInterval=5
StartLimitBurst=10
ExecStart=/bin/sh -c '[ -n "${CRI_SOCKET}" ] && exec /usr/local/bin/k0s controller --config=/etc/k0s/k0s.yaml --cri-socket=${CRI_SOCKET} --token-file=/etc/k0s/controller-token|| exec /usr/local/bin/k0s controller --config=/etc/k0s/k0s.yaml --token-file=/etc/k0s/controller-token'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Who's supposed to generate this /etc/k0s/controller-token ?

Should we start the k0s.service before ? If yes, it should be documented.


RestartSec=10
Delegate=yes
KillMode=process
LimitCORE=infinity
TasksMax=infinity
TimeoutStartSec=0
LimitNOFILE=999999
Restart=always

[Install]
WantedBy=multi-user.target
EOF

cat > "${SYSEXTNAME}"/usr/local/lib/systemd/system/k0sworker.service << EOF
[Unit]
Description=k0s - Worker
Documentation=https://docs.k0sproject.io
ConditionFileIsExecutable=/usr/local/bin/k0s

Requires=containerd.service
Wants=network-online.target
After=network-online.target containerd.service

[Service]
EnvironmentFile=-/etc/default/k0s
StartLimitInterval=5
StartLimitBurst=10
ExecStart=/usr/local/bin/k0s worker --cri-socket=$CRI_SOCKET --token-file=/etc/k0s/worker-token
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
ExecStart=/usr/local/bin/k0s worker --cri-socket=$CRI_SOCKET --token-file=/etc/k0s/worker-token
ExecStart=/usr/local/bin/k0s worker --cri-socket="${CRI_SOCKET}" --token-file=/etc/k0s/worker-token

ExecStart=/bin/sh -c '[ -n "${CRI_SOCKET}" ] && exec /usr/local/bin/k0s worker --cri-socket=${CRI_SOCKET} --token-file=/etc/k0s/worker-token|| exec /usr/local/bin/k0s worker --token-file=/etc/k0s/worker-token'

RestartSec=10
Delegate=yes
KillMode=process
LimitCORE=infinity
TasksMax=infinity
TimeoutStartSec=0
LimitNOFILE=999999
Restart=always

[Install]
WantedBy=multi-user.target
EOF

RELOAD=1 "${SCRIPTFOLDER}"/bake.sh "${SYSEXTNAME}"
rm -rf "${SYSEXTNAME}"