K8S cluster on Raspberry Pi

Project

This project allows everyone to easily build a Kubernetes cluster using Raspberry Pi and Rancher's k3s.

The final product is a K3S cluster with etcd in HA running on Raspberry Pi boards with USB3.0 attached SSDs.

K3S will use:

• calico as CNI
• (optional) Metallb as load balancer
• (optional) ingress-nginx as ingress
• (optional) CSIs
• (optional) cert-manager

Purpose

I initially created it for my personal use, from experimenting with cloudnative-pg and longhorn, to run my own cloud services using nextcloud.

The main purpose is to share the gained knowledge and the collection of bash scripts with anyone is interested in learning, practicing, and improving this project. My recommendation is to first follow the procedure and get confidence with the commands by manually apply them step by step. Then, one could always recreate the entire cluster and the deployed resources in a semi-automatic fashion using the provided scripts.

The scripts are mainly a solution to speed up the entire setup process from scratch, and ensuring reproducibility.

Hardware

Minimum Requirements

• x3 RPi 8GB RAM
• x3 SD cards
• x3 SSD disks
• x3 SATA to USB-3
• x1 4 ports network switch

Power adapters/cables, and network cables are omitted, but required.

Tested

• Boards: Raspberry Pi 4 8GB RAM
• Power: RPi PoE HAT or USB power adapters
• SATA to USB3.0 adapter: UGREEN with UASP TRIM support
• SSD: Samsung 860 EVO - 1TB
• Network Switch: TP-Link SG1005P (PoE)

Distribution

• Debian for RPi

Tested on Debian 12 Bookworm for arm64.

Procedure

Follow the procedure in the docs/ directory:

0. Update RPi4 Firmware (not necessary anymore)
1. Install Debian for RPi
2. Clone image on SSD and mount it as root
3. Prepare RPi nodes installing and configuring packages
4. Setup cluster installing k3s on each node
5. Install additional resources

Scripts

Scripts Overview

Procedural

1. install-sd.sh - to write distro image into SD card, add SSH key, configure hostname, download rpi-clone repo inside /root
2. clone-root-to-ssd.sh - to automate the procedure to clone SD to SSD
3. install-kluster.sh - to install k3s and deploy required resources on the cluster following the correct procedure
4. deploy-resources.sh (optional) - to deploy the specified kubernetes resource(s) from a list or just one resource

Others

• intall-k3s-master.sh - to install and deploy the k3s master (init and join)
• join-k3s-agent.sh - to install and deploy a new k3s agent to join the cluster
• enable-fstrim.sh - to enable fstrim on target nodes where supported

Tests

We are here to experiment and learn, so let's test things out!

• PostgreSQL Resiliency test

Sources

Raspberry Pi

• Update Raspberry Firmware #1
• Raspberry Firmware Release Notes
• Boot from USB
• Debian for RPi
• Image clone guide
• RPi Clone tool
• RPi Boot from SSD
• Raspberry Pi cheat sheet
• SSD TRIM

Kubernetes

• K8S on RPi video
• K8S on RPi
• K3S multimaster HA

Network

• Calico installation

Load Balancer

• MetalLB on GH

Storage

• Longhorn on GH
• TopoLVM on GH

Ingress

• Nginx Ingress on GH

Database

• Cloudnative-pg on GH