28 Aug 2026 · 10 min read

Pi Cluster Rebuild Guide

This is my reference for rebuilding my five-node Raspberry Pi 4 k3s cluster from scratch.

The cluster consists of five Raspberry Pi 4s with 8 GB of RAM each: one control-plane node and four workers. It runs Ubuntu Server 26.04 LTS, k3s, Rancher, Helm, and cert-manager.

I'm documenting the entire process here mostly so the next time I inevitably break or rebuild the cluster, I don't have to figure everything out again.

Raspberry Pi 4 Five-Node Rack Mounted Cluster

Nodes:

Hostname	IP	Role
k3s-master	xxx.xx.x.80	control-plane
k3s-worker-01	xxx.xx.x.81	worker
k3s-worker-02	xxx.xx.x.82	worker
k3s-worker-03	xxx.xx.x.83	worker
k3s-worker-04	xxx.xx.x.84	worker

Use the same username on all nodes: ubuntu or whatever you set during imaging (e.g., jriv)

1. Image the OS (per node)

Download Raspberry Pi Imager from raspberrypi.com/software
Insert SD card / storage for the node
Choose OS → Other general-purpose OS → Ubuntu → Ubuntu Server 26.04 LTS (64-bit)
Choose Storage → select the correct device.
Click the gear/advanced settings icon before writing:
- Set unique hostname (k3s-master, k3s-worker-01, k3s-worker-02, k3s-worker-03, k3s-worker-04, etc.)
- Set username: ubuntu
- Enable SSH, add your public key or set a password for the ubuntu user
- Set WiFi/locale if not using Ethernet
Write the image
Repeat for each node, changing only the hostname

2. Prepare Each Node

SSH in (by IP if .local hostname resolution isn't working, find the node's IP address in your router's client list and connect directly.):

ssh ubuntu@<node-ip>

Update the system:

sudo apt update && sudo apt upgrade -y

On a fresh Ubuntu install, unattended-upgrades may already be running and temporarily lock apt. If that happens, let it finish before continuing. You can check progress with:

sudo tail -f /var/log/unattended-upgrades/unattended-upgrades.log

Disable unattended-upgrades (avoid future collisions with manual work):

I disable unattended upgrades on these nodes because I prefer to control when package updates happen and avoid nodes updating independently while the cluster is running.

sudo systemctl stop unattended-upgrades
sudo systemctl disable unattended-upgrades

USB boot readiness (do this now, even if you will be booting from SD today)

sudo apt install rpi-eeprom -y
sudo rpi-eeprom-update
sudo rpi-eeprom-config --edit

In the editor, find or add under [all]:

BOOT_ORDER=0xf41

This tries SD first, falls back to USB, retries, flag list may vary slightly by Pi 4 revision, that's normal.

Save and exit (Ctrl+O, Enter, Ctrl+X).

Reboot to apply both the kernel update and EEPROM change:

sudo reboot

Confirm after reboot:

uname -r        # updated kernel
hostname        # correct hostname

Note: if a node's hostname reverts after reboot, the real source is /boot/firmware/user-data (cloud-init), not just hostnamectl. Edit the hostname: line there directly, then also run sudo hostnamectl set-hostname <name> and reboot again.

3. Install k3s

On master ONLY:

curl -sfL https://get.k3s.io | sh -

Confirm it's running:

sudo systemctl status k3s
sudo k3s kubectl get nodes

Get the join token:

sudo cat /var/lib/rancher/k3s/server/node-token

On each worker (NOT master):

curl -sfL https://get.k3s.io | K3S_URL=https://<master-ip>:6443 K3S_TOKEN=<paste-token> sh -

Verify from master:

kubectl get nodes -o wide

All nodes should show Ready.

4. Configure kubectl for the Ubuntu User

Note: I use Zsh on my nodes, so the examples below use ~/.zshrc. If you're using Ubuntu's default Bash shell, use ~/.bashrc instead.

mkdir -p ~/.kube
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
sudo chown $(id -u):$(id -g) ~/.kube/config

export KUBECONFIG=~/.kube/config
echo 'export KUBECONFIG=~/.kube/config' >> ~/.zshrc
source ~/.zshrc

5. Install Helm + cert-manager

curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
helm version

helm repo add jetstack https://charts.jetstack.io
helm repo update
kubectl create namespace cert-manager
helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --set crds.enabled=true

Confirm all 3 cert-manager pods are Running:

kubectl get pods --namespace cert-manager

6. Local DNS for Rancher

I'm using a UDM/Dream Machine in my homelab. This is how I set up local DNS for rancher.lan on the UDM/Dream Machine: On UDM/Dream Machine: Network app → Settings → DNS/Policy → Create Policy

Type: DNS
Rule type: Host (A)
Domain Name: rancher.lan // replace with your desired local domain name
IP Address: xxx.xx.x.xx (master)
TTL: Auto

7. Install Rancher

kubectl create namespace cattle-system

helm repo add rancher-latest https://releases.rancher.com/server-charts/latest
helm repo update

helm install rancher rancher-latest/rancher \
  --namespace cattle-system \
  --set hostname=rancher.lan \
  --set bootstrapPassword=<temporary-bootstrap-password>

Replace with a temporary password of your choice. You'll use it for the initial Rancher login and replace it with a permanent password afterward.

Watch rollout:

kubectl -n cattle-system get pods -w

Press Ctrl+C to exit watch mode. This doesn't stop anything; it only stops streaming the pod status to your terminal.

Known issue on Pi hardware: startup probe timeout

Rancher's default startup probe only allows ~2 minutes to come up, which can be too short on Pi 4 CPU speed (especially during the initial Helm chart catalog restore). If pods keep restarting with Error / exit code 137 and logs show it's still mid-catalog-restore, patch the probe:

kubectl -n cattle-system patch deployment rancher --type='json' -p='[
  {"op": "replace", "path": "/spec/template/spec/containers/0/startupProbe/failureThreshold", "value": 60}
]'

This extends the window to 10 minutes. Watch pods again, a new ReplicaSet spins up with the patched setting. Old pods terminate as replacements come up. Some Terminating → Error transitions during this are normal/cosmetic, not new failures.

Once kubectl -n cattle-system rollout status deploy/rancher reports success and all replicas show 1/1 Running, log into https://rancher.lan with the bootstrap password and set a real password immediately.

Note: joining a new node later does NOT automatically move/add Rancher replicas to it — Kubernetes only schedules on pod creation, not retroactively. Scale replicas or manually delete/recreate a pod if you want it rebalanced.

Rancher running on the finished five-node Raspberry Pi k3s cluster.

8. SSH Key Setup (Passwordless Login)

Passwordless SSH is needed for the shutdown script below (and is generally more convenient than entering a password every time you connect to a node).

Check for an existing SSH key on your Mac:

ls -la ~/.ssh/

Look for a matching key pair, such as id_ed25519 / id_ed25519.pub or id_rsa / id_rsa.pub. If one already exists, you can skip generating a new one.

If you don't have one, generate an Ed25519 key:

ssh-keygen -t ed25519

Press Enter to accept the default save location. Leave the passphrase blank (press Enter twice) if you want scripts to use the key unattended without relying on ssh-agent to cache the passphrase.

Copy the public key to each of the five nodes. You'll be prompted for each node's password one last time:

ssh-copy-id -i ~/.ssh/id_ed25519.pub ubuntu@<master-ip>
ssh-copy-id -i ~/.ssh/id_ed25519.pub ubuntu@<worker-01-ip>
ssh-copy-id -i ~/.ssh/id_ed25519.pub ubuntu@<worker-02-ip>
ssh-copy-id -i ~/.ssh/id_ed25519.pub ubuntu@<worker-03-ip>
ssh-copy-id -i ~/.ssh/id_ed25519.pub ubuntu@<worker-04-ip>

If you're using a different key, replace the -i path with the appropriate public key, such as ~/.ssh/id_rsa.pub.

Test the connection:

ssh ubuntu@<master-ip>

You should connect without being prompted for the node's password. Once this works on all five nodes, the SSH keys combined with the passwordless shutdown sudo rule below allow clusterdown to shut down the entire cluster without prompting for credentials.

9. Shutdown script (Mac or Linux)

I don't keep my cluster on 24/7, so I use a shutdown script to power it down cleanly when not in use. Shut workers down first, master last, so it closes cleanly.

One-time setup per node — passwordless sudo for shutdown only:

echo "ubuntu ALL=(ALL) NOPASSWD: /usr/sbin/shutdown" | sudo tee /etc/sudoers.d/ubuntu-shutdown

Script — save as ~/cluster-shutdown.sh:

#!/bin/bash
WORKERS="<worker-01-ip> <worker-02-ip> <worker-03-ip> <worker-04-ip>"
MASTER="<master-ip>"

echo "Shutting down cluster..."

for ip in $WORKERS; do
  echo "→ Shutting down worker $ip..."
  ssh ubuntu@$ip "sudo shutdown -h now"
  echo "  done, waiting 5s..."
  sleep 5
done

echo "→ Shutting down master $MASTER..."
ssh ubuntu@$MASTER "sudo shutdown -h now"
echo "Cluster shutdown complete."

chmod +x ~/cluster-shutdown.sh

Optional alias (add to ~/.zshrc or ~/.bashrc if using Zsh or Bash):

alias clusterdown="~/cluster-shutdown.sh"

Startup: power everything back on in any order, workers will retry connecting to master until it's up, self-healing within a minute or two.

Troubleshooting notes

If you accidentally ran the plain curl ... | sh - install command on a worker, it installs a second server instead of joining as an agent. Uninstall it with:
```
sudo /usr/local/bin/k3s-uninstall.sh
```
Then reinstall using the correct K3S_URL / K3S_TOKEN command for joining the cluster (example: K3S_URL=https://<master-ip>:6443 K3S_TOKEN=<token> sh -).