A tour of the homelab
The design rules behind a Proxmox, OPNsense and microk8s lab: one small node keeps the basics running, the heavy nodes sleep until needed, and nothing replicates twice.
The lab is built around one rule: if the smallest box in the rack is up, the basics still work: network, DNS, certificates.
Everything else follows from that. The family needs DNS, a router, photos and file sync. I need a Kubernetes cluster that can chew on a stream-processing job for a weekend and then power off. Those two requirements pull in opposite directions, and the layout of the lab is how I split them.
Layer 1: Proxmox, with a bootstrap node
The hypervisor layer is a Proxmox VE cluster. Racked for eleven hosts, eight currently joined. They are not a uniform fleet, and that is deliberate.
pve-n1 is the bootstrap host: the smallest box in the cluster and the only machine on a UPS. It carries what has to exist before anything else can start:
- the router VM
- authoritative DNS
- the internal certificate authority
- PostgreSQL
- a storage container for cold data
- backup storage
Small, but nothing on it is optional. If pve-n1 is up, name resolution works, certificates can be issued, and the network routes. Everything above it is allowed to be down.
pve-n2 through n5 host the always-on part of the cluster. Each runs a VM that is a microk8s node, control plane and worker at once, carrying the household apps (file sync, media, photos) and the microservices for my personal projects. pve-n6 through n11 host the heavy workers for data processing, and they are powered off unless a job needs them.
That last group is the part that surprises people. Idle hardware costs power and noise, and the workload that justifies it is bursty. Starting a node on demand is a few minutes of latency I’m happy to pay.
Layer 2: the router is a VM
OPNsense runs as a VM on pve-n1. It handles routing, DHCP, ad filtering through Unbound, and TLS termination in front of the cluster. Unbound forwards my own zones to the authoritative server, so internal names resolve the same way from any client.
Putting the router in a VM is a trade-off, not a free lunch. The blast radius of a hypervisor failure now includes the network. I accept it because pve-n1 is the one node I protect: UPS, minimal workload, no experiments. The lab’s blast radius is wherever I let myself experiment, and that is never on n1.
Layer 3: core services on the bootstrap node
Above the router sit the shared services that everything else consumes: a PostgreSQL instance, an internal CA issuing certificates over ACME, and a small Docker host for the underlay (reverse proxy, SSO, secrets, git, metrics and logs).
That PostgreSQL instance has a narrow job: it holds state for the core services, the things that have to work before Kubernetes does, like DNS. It is deliberately not where I experiment. Application databases live in the cluster, on CloudNativePG, where I can break them, rebuild them and tear them down without touching the house.
That boundary is still settling. The photo and file-sync services currently keep their data on the core instance, a leftover from before the cluster had a proper Postgres operator. Now that it does, I expect them to move, which leaves the core instance smaller and more boring. That’s the goal. The backup runs as a Kubernetes CronJob that dumps to object storage, and a small dump is one I’ll actually test a restore from.
Layer 4: Kubernetes on microk8s
The cluster used to run on Rancher. I tore it down and rebuilt on microk8s, with every node a VM on a Proxmox host. The four on pve-n2 to n5 are control plane and worker at once. The heavy workers on pve-n6 to n11 take the data-processing jobs. Storage is Longhorn, with each virtual disk backed by its own physical SSD.
The rule I hold hardest here: no stacked replication. If the application replicates itself, as a database does, the Longhorn volume gets a single replica. Normal volumes get two. I do not put a replicated storage layer under another replicated storage layer, because you pay twice in write latency and get one failure domain’s worth of resilience.
Backups go to an NFS share on the storage container, not into the cluster. That is why restoring volumes from the old Rancher cluster into the new one was possible at all.
Cluster add-ons are managed with Terraform. Where there was a choice between an add-on and plain Helm, I took Helm. It’s one more thing to own, but I know what it installed.
Telemetry and the AI angle
Metrics and logs land in VictoriaMetrics and VictoriaLogs, with Grafana on top. The stack runs on the Docker underlay host on pve-n1, so telemetry stays up whenever the bootstrap node does, including while the cluster is down and I most want to see why. A gateway exposes those backends over MCP so an assistant can query them directly. Most of the troubleshooting and infrastructure-as-code in this lab now happens in a conversation with Claude, which is also where the material for this section comes from.
What this doesn’t handle
The single UPS is the honest weakness. The four control-plane nodes live on pve-n2 to n5, none of them on the UPS, so a power event takes out quorum while the router and DNS stay up on pve-n1. The network, DNS and certificates keep working; the cluster doesn’t. That’s the trade I made for the design rule above, and I’d like to revisit it.