-
Notifications
You must be signed in to change notification settings - Fork 40
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bake k0s as a systemd sysext image. #99
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -91,6 +91,7 @@ For extensions that are not part of the GitHub Release or which you want to cust | |
| `wasmcloud` | released | | ||
| `tailscale` | released | | ||
| `crio` | released | | ||
| `k0s` | released | | ||
| `k3s` | released | | ||
| `rke2` | released | | ||
| `keepalived` | build script | | ||
|
@@ -223,6 +224,114 @@ storage: | |
Of course its also possible to use the | ||
[artifact-follower](https://falco.org/blog/falcoctl-install-manage-rules-plugins/#follow-artifacts) to download falco artifacts automatically. | ||
|
||
#### k0s | ||
|
||
To setup [k0s](https://docs.k0sproject.io/stable/) we need: | ||
|
||
1. the sysext plus the configuration files /etc/k0s/k0s.yaml and/or /etc/default/k0s | ||
2. the systemd units | ||
|
||
After you enable k0s syext following the guide deacribed on Consuming the published images, follow the next steps: | ||
|
||
1. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is this mandatory? One might not know by advance the private IP address. |
||
a) Customize /etc/k0s/k0s.yaml for either controller/worker ([More info](https://docs.k0sproject.io/stable/configuration/#configuration-file-reference)). Below is the default configuration for k0s implemented via Ignition which you should change it according to your needs: | ||
|
||
```yaml | ||
storage: | ||
files: | ||
- path: /etc/k0s/k0s.yaml | ||
overwrite: true | ||
mode: 0644 | ||
contents: | ||
inline: | | ||
apiVersion: k0s.k0sproject.io/v1beta1 | ||
kind: ClusterConfig | ||
metadata: | ||
name: k0s | ||
spec: | ||
api: | ||
address: 192.168.68.104 | ||
k0sApiPort: 9443 | ||
port: 6443 | ||
sans: | ||
- 192.168.68.104 | ||
controllerManager: {} | ||
extensions: | ||
helm: | ||
concurrencyLevel: 5 | ||
installConfig: | ||
users: | ||
etcdUser: etcd | ||
kineUser: kube-apiserver | ||
konnectivityUser: konnectivity-server | ||
kubeAPIserverUser: kube-apiserver | ||
kubeSchedulerUser: kube-scheduler | ||
konnectivity: | ||
adminPort: 8133 | ||
agentPort: 8132 | ||
network: | ||
clusterDomain: cluster.local | ||
dualStack: | ||
enabled: false | ||
kubeProxy: | ||
iptables: | ||
minSyncPeriod: 0s | ||
syncPeriod: 0s | ||
ipvs: | ||
minSyncPeriod: 0s | ||
syncPeriod: 0s | ||
tcpFinTimeout: 0s | ||
tcpTimeout: 0s | ||
udpTimeout: 0s | ||
metricsBindAddress: 0.0.0.0:10249 | ||
mode: iptables | ||
kuberouter: | ||
autoMTU: true | ||
hairpin: Enabled | ||
metricsPort: 8080 | ||
nodeLocalLoadBalancing: | ||
enabled: false | ||
envoyProxy: | ||
apiServerBindPort: 7443 | ||
konnectivityServerBindPort: 7132 | ||
type: EnvoyProxy | ||
podCIDR: 10.244.0.0/16 | ||
provider: kuberouter | ||
serviceCIDR: 10.96.0.0/12 | ||
scheduler: {} | ||
storage: | ||
etcd: | ||
peerAddress: 192.168.68.104 | ||
type: etcd | ||
telemetry: | ||
enabled: true | ||
``` | ||
|
||
b) If you want use another CRI other than k0s embedded one define it here: /etc/default/k0s | ||
|
||
```yaml | ||
storage: | ||
files: | ||
- path: /etc/default/k0s | ||
overwrite: true | ||
mode: 0644 | ||
contents: | ||
inline: | | ||
CRI_SOCKET="remote:unix:///var/run/custom_CRI.sock" | ||
``` | ||
|
||
2. Defined systemd units are: k0s.service, k0scontroller.service and k0sworker.service | ||
a) If you go with the embedded k0s components you have to enable k0scontroller.service or k0sworker.service according to the `ROLE`(controller/worker) of the node and the /etc/k0s/k0s.yaml you have defined at step 1 | ||
|
||
```yaml | ||
systemd: | ||
units: | ||
- name: k0s`ROLE`.service | ||
enabled: true | ||
``` | ||
|
||
b) If you decide to go with custom components for k0s (different CRI, external ETCD cluster, ...) you have to enable k0s.service for controller or k0sworker.service for worker and, of course, configure /etc/k0s/k0s.yaml accordingly. | ||
|
||
#### Kubernetes | ||
|
||
The [Flatcar Kubernetes docs](https://www.flatcar.org/docs/latest/container-runtimes/getting-started-with-kubernetes/) show how to use the extension provided here for controllers and workers. | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This file needs to be pushed with executable mode: diff --git a/create_k0s_sysext.sh b/create_k0s_sysext.sh
old mode 100644
new mode 100755 |
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
@@ -0,0 +1,130 @@ | ||||||
#!/usr/bin/env bash | ||||||
set -euo pipefail | ||||||
|
||||||
export ARCH="${ARCH-x86-64}" | ||||||
SCRIPTFOLDER="$(dirname "$(readlink -f "$0")")" | ||||||
|
||||||
if [ $# -lt 2 ] || [ "$1" = "-h" ] || [ "$1" = "--help" ]; then | ||||||
echo "Usage: $0 VERSION SYSEXTNAME" | ||||||
echo "The script will download the k0s binary (e.g., for v1.31.2+k0s.0) and create a sysext squashfs image with the name SYSEXTNAME.raw in the current folder." | ||||||
echo "A temporary directory named SYSEXTNAME in the current folder will be created and deleted again." | ||||||
echo "All files in the sysext image will be owned by root." | ||||||
echo "To use arm64 pass 'ARCH=arm64' as environment variable (current value is '${ARCH}')." | ||||||
"${SCRIPTFOLDER}"/bake.sh --help | ||||||
exit 1 | ||||||
fi | ||||||
|
||||||
VERSION="$1" | ||||||
SYSEXTNAME="$2" | ||||||
|
||||||
# The github release uses different arch identifiers, we map them here | ||||||
# and rely on bake.sh to map them back to what systemd expects | ||||||
if [ "${ARCH}" = "amd64" ] || [ "${ARCH}" = "x86-64" ]; then | ||||||
ARCH="amd64" | ||||||
fi | ||||||
if [ "${ARCH}" = "arm64" ] || [ "${ARCH}" = "aarch64" ]; then | ||||||
ARCH="arm64" | ||||||
fi | ||||||
|
||||||
URL="https://github.com/k0sproject/k0s/releases/download/${VERSION}/k0s-${VERSION}-${ARCH}" | ||||||
|
||||||
rm -rf "${SYSEXTNAME}" | ||||||
mkdir -p "${SYSEXTNAME}"/usr/local/bin | ||||||
curl -o "${SYSEXTNAME}/usr/local/bin/k0s" -fsSL "${URL}" | ||||||
chmod +x "${SYSEXTNAME}"/usr/local/bin/k0s | ||||||
pushd "${SYSEXTNAME}"/usr/local/bin/ | ||||||
ln -s ./k0s kubectl | ||||||
ln -s ./k0s ctr | ||||||
popd | ||||||
|
||||||
mkdir -p "${SYSEXTNAME}"/usr/local/lib/systemd/system/ | ||||||
cat > "${SYSEXTNAME}"/usr/local/lib/systemd/system/k0s.service << EOF | ||||||
[Unit] | ||||||
Description=k0s - Init Controller / External ETCD Controller | ||||||
Documentation=https://docs.k0sproject.io | ||||||
ConditionFileIsExecutable=/usr/local/bin/k0s | ||||||
|
||||||
Requires=containerd.service | ||||||
Wants=network-online.target | ||||||
After=network-online.target containerd.service | ||||||
|
||||||
[Service] | ||||||
EnvironmentFile=-/etc/default/k0s | ||||||
StartLimitInterval=5 | ||||||
StartLimitBurst=10 | ||||||
ExecStart=/bin/sh -c '[ -n "${CRI_SOCKET}" ] && exec /usr/local/bin/k0s controller --config=/etc/k0s/k0s.yaml --cri-socket=${CRI_SOCKET} || exec /usr/local/bin/k0s controller --config=/etc/k0s/k0s.yaml' | ||||||
tormath1 marked this conversation as resolved.
Show resolved
Hide resolved
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This won't work:
You need to escape those values (individually or globally with There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
(Same for the other units) |
||||||
|
||||||
RestartSec=10 | ||||||
Delegate=yes | ||||||
KillMode=process | ||||||
LimitCORE=infinity | ||||||
TasksMax=infinity | ||||||
TimeoutStartSec=0 | ||||||
LimitNOFILE=999999 | ||||||
Restart=always | ||||||
|
||||||
[Install] | ||||||
WantedBy=multi-user.target | ||||||
EOF | ||||||
|
||||||
cat > "${SYSEXTNAME}"/usr/local/lib/systemd/system/k0scontroller.service << EOF | ||||||
[Unit] | ||||||
Description=k0s - Controller | ||||||
Documentation=https://docs.k0sproject.io | ||||||
ConditionFileIsExecutable=/usr/local/bin/k0s | ||||||
|
||||||
Requires=containerd.service | ||||||
Wants=network-online.target | ||||||
After=network-online.target containerd.service | ||||||
|
||||||
[Service] | ||||||
EnvironmentFile=-/etc/default/k0s | ||||||
StartLimitInterval=5 | ||||||
StartLimitBurst=10 | ||||||
ExecStart=/bin/sh -c '[ -n "${CRI_SOCKET}" ] && exec /usr/local/bin/k0s controller --config=/etc/k0s/k0s.yaml --cri-socket=${CRI_SOCKET} --token-file=/etc/k0s/controller-token|| exec /usr/local/bin/k0s controller --config=/etc/k0s/k0s.yaml --token-file=/etc/k0s/controller-token' | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Should we start the |
||||||
|
||||||
RestartSec=10 | ||||||
Delegate=yes | ||||||
KillMode=process | ||||||
LimitCORE=infinity | ||||||
TasksMax=infinity | ||||||
TimeoutStartSec=0 | ||||||
LimitNOFILE=999999 | ||||||
Restart=always | ||||||
|
||||||
[Install] | ||||||
WantedBy=multi-user.target | ||||||
EOF | ||||||
|
||||||
cat > "${SYSEXTNAME}"/usr/local/lib/systemd/system/k0sworker.service << EOF | ||||||
[Unit] | ||||||
Description=k0s - Worker | ||||||
Documentation=https://docs.k0sproject.io | ||||||
ConditionFileIsExecutable=/usr/local/bin/k0s | ||||||
|
||||||
Requires=containerd.service | ||||||
Wants=network-online.target | ||||||
After=network-online.target containerd.service | ||||||
|
||||||
[Service] | ||||||
EnvironmentFile=-/etc/default/k0s | ||||||
StartLimitInterval=5 | ||||||
StartLimitBurst=10 | ||||||
ExecStart=/usr/local/bin/k0s worker --cri-socket=$CRI_SOCKET --token-file=/etc/k0s/worker-token | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
ExecStart=/bin/sh -c '[ -n "${CRI_SOCKET}" ] && exec /usr/local/bin/k0s worker --cri-socket=${CRI_SOCKET} --token-file=/etc/k0s/worker-token|| exec /usr/local/bin/k0s worker --token-file=/etc/k0s/worker-token' | ||||||
|
||||||
RestartSec=10 | ||||||
Delegate=yes | ||||||
KillMode=process | ||||||
LimitCORE=infinity | ||||||
TasksMax=infinity | ||||||
TimeoutStartSec=0 | ||||||
LimitNOFILE=999999 | ||||||
Restart=always | ||||||
|
||||||
[Install] | ||||||
WantedBy=multi-user.target | ||||||
EOF | ||||||
|
||||||
RELOAD=1 "${SCRIPTFOLDER}"/bake.sh "${SYSEXTNAME}" | ||||||
rm -rf "${SYSEXTNAME}" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.