skip to content
BitsAndBytes
Table of Contents

Part 1 built the GitOps loop. Part 2 gave it DNS, TLS, secrets and monitoring, and ended with a claim: the platform absorbs new services for free. Part 3 is that claim being stress-tested, because in the two days since, the cluster ate half my homelab:

  • 🪦 the monitoring VM — deleted, its job absorbed by the cluster
  • 📺 the entire *arr media stack — nine services, live-migrated with their state, behind a VPN
  • 🌐 every reverse-proxy route in the house — moved from clicked-together NPM rules to YAML in Git
  • 🔍 plus two services and a Cloudflare tunnel that turned out to be serving nobody

The first victim: a VM the cluster made redundant

The old monitoring VM ran Prometheus and Grafana for the docker fleet. Part 2’s kube-prometheus-stack does that job better — so the only thing keeping that VM alive was inertia. The checklist to kill it: migrate the one irreplaceable workload (pve-exporter, which watches the Proxmox hypervisor itself — the thing that knows my RAM headroom), point the new Prometheus at the remaining node_exporters on the other VMs, archive the old Grafana’s database, qm destroy.

The exporter migration was the part 2 pattern working exactly as advertised: API token → 1Password → ExternalSecret → pod env, plus my first ServiceMonitor — and Prometheus discovered it with zero configuration changes. The external VMs needed the third member of the discovery family, a ScrapeConfig (“scrape these static LAN addresses”), which also needed the chart’s third horribly-named selector flag set to false. There’s a pattern here: if a Prometheus Operator feature mysteriously does nothing, search the values for ...SelectorNilUsesHelmValues.

The deletion itself produced the best war story of the week. qm destroy 102 --purge removed the VM but ZFS refused to release its disk: dataset is busy. The forensic trail, peeled one layer at a time:

flowchart TB
Z["💾 zfs destroy vm-102-disk-0<br/>'dataset is busy'"] --> N["🔍 qemu-nbd process<br/>started Dec 24, 2025<br/>exporting the disk to /dev/nbd0"]
N --> L["🔍 host LVM auto-activated<br/>the guest's ubuntu-vg<br/>through that export"]
L --> P["🔍 a loop device<br/>on partition 3,<br/>also from December"]
P --> D["✅ disconnect · deactivate · detach<br/>dataset destroyed"]

Some December-me had inspected this VM’s disk with qemu-nbd, mounted its LVM, and never cleaned up. Six months later the kernel was still faithfully holding all three layers open. If zfs destroy ever tells you a dead VM’s dataset is busy: check qemu-nbd processes, dmsetup ls, and losetup -a before you doubt your sanity.

Refund: 4 GiB of RAM on a hypervisor where RAM is the binding constraint, 22 GB of pool space, one fewer VM to patch.

The hard problem: an *arr stack that must stay behind one VPN

The media stack — sonarr, radarr, prowlarr, bazarr, jellyseerr, qbittorrent, sabnzbd, flaresolverr, unpackerr — ran as one docker-compose file where every container shared the VPN container’s network namespace (network_mode: "container:gluetun"). One tunnel, one egress IP, everything behind it. My constraint for the migration was simple: keep exactly that.

Kubernetes has four answers to “pods behind a VPN”, and researching them was a lesson in trade-offs:

  1. One mega-pod — a pod is a shared network namespace, so gluetun-as-sidecar plus nine app containers is a faithful clone. But a pod is also the unit of updates: every Renovate merge would restart the entire stack. Faithful, and infuriating.
  2. Gluetun sidecar per app — what most of the community does, but they only put the downloader behind the VPN. Nine sidecars means nine tunnels (providers cap connections) with nine different IPs. Fails the constraint.
  3. Routing-layer tricks (CNI subnets + policy routing on the router) — elegant, infrastructure-wide, and requires replacing the CNI and teaching the UniFi gateway policy NAT. Admire from a distance.
  4. pod-gateway — one gateway pod runs gluetun; an admission webhook injects a tiny client into any pod born in a marked namespace, routing its traffic to the gateway over a VXLAN. One tunnel, one IP, and independent per-app updates.
flowchart TB
POD["📦 new pod in 'media' namespace"] -- "webhook injects client" --> VX["🕸️ VXLAN tunnel<br/>inside the cluster"]
VX --> GW["🚪 gateway pod<br/>pod-gateway + gluetun"]
GW -- "WireGuard, one tunnel" --> VPN["🔒 VPN provider<br/>one egress IP for all"]
GW -. "gateway down?<br/>client has NO internet<br/>(kill-switch)" .-> DEAD["⛔"]

Option 4 won. Two k3s-specific landmines for anyone following: the webhook triggers on a namespace label (routed-gateway: "true"), not just the chart’s routed_namespaces list — and the injected client deletes the pod’s default route before resolving the gateway’s address, which strands cluster DNS unless NOT_ROUTED_TO_GATEWAY_CIDRS includes the service CIDR (flannel only routes the pod CIDR). Symptoms: no injection at all, then network unreachable in init.

The acceptance tests are the fun part. A throwaway pod in the namespace egressed with the VPN provider’s Amsterdam IP, not my home address. Scaling the gateway to zero made the client lose the internet entirely — fail closed, never leaking the home IP. And a 100 MB pull through VXLAN-over-WireGuard ran at ~245 Mbps. Clone proven, mediahub untouched.

The big bang: nine services in an afternoon

With the gateway proven, caution lost its appeal and we migrated everything at once. The choreography:

flowchart TB
F["🧊 freeze: stop the 9 old containers<br/>(SQLite quiesced, exact snapshot kept)"] --> T["📦 transplant: tar each /opt/appdata<br/>over SSH into its PVC"]
T --> U["🔌 rewire: localhost URLs →<br/>cluster service names<br/>(text configs + *arr APIs)"]
U --> V["✅ verify: UIs, health checks,<br/>downloads, Telegram intact"]
V -.-> R["↩️ rollback any time:<br/>docker start on the old VM"]

The configs are the apps — SQLite databases and XML carrying every quality profile, notification hook and API key. Copy those, keep the container paths identical (/config, /data), and the new pods wake up believing they are the old containers. Media itself never moved: it lives on a Synology NFS export that the cluster simply mounts too.

What actually went wrong, in order: the k3s VM lacked nfs-common (cloud images are minimal); the Synology export ACL didn’t know the new client IP; and then the good one — pods could see the share but not traverse it. The directory was 770, owned by UID 1031. My manifests ran everything as the conventional UID 1000… but the old stack never did. Buried in the old .env: PUID=1031 — the Synology user who owns the share. The containers had been impersonating the NAS owner all along. Two lines changed, and years of accumulated state worked perfectly: Telegram notifications, download history, indexer configs, all of it.

Smaller bruises worth recording: apps that lived in one network namespace reference each other as localhost, and those URLs hide in SQLite, so the *arr download clients and indexers needed repointing through their APIs — where ?forceSave=true skips the save-time connection test, and piping a PUT response through head kills the request mid-flight via SIGPIPE (ask me how I know). Compose port mappings (8200:49790) become real listen ports in k8s, so two services needed their actual ports looked up. And pin your images by digest-matching what’s running: my unpackerr pin guessed a tag that didn’t exist, which led to discovering the hotio images dropped semver tags entirely — switched to the official image, which Renovate can actually track.

Jellyfin deliberately stayed on the old VM: it holds the iGPU via PCIe passthrough for hardware transcoding, and a passed-through device belongs to one VM. That’s a future decision, not a blocker — the hybrid end-state from part 1 doing its job.

The quiet revolution: every route into Git

The last act was less dramatic and maybe more consequential. Nginx Proxy Manager held 28 clicked-together proxy routes — the accumulated routing table of the whole homelab. Kubernetes has a clean pattern for “route to something outside the cluster”: a Service with no selector plus a hand-written EndpointSlice pointing at a LAN IP, fronted by a normal Ingress.

flowchart TB
B["💻 proxmox.huislab.app"] --> U["🧭 Unbound<br/>per-name redirect"]
U --> T["🌐 traefik<br/>Ingress (in Git) + wildcard cert"]
T --> EP["🔌 selector-less Service<br/>+ EndpointSlice: 192.168.1.7:8006"]
EP --> PVE["🖥️ Proxmox<br/>(stays exactly where it is)"]

Twelve routes migrated this way — Proxmox, Home Assistant, the NAS, UniFi, Jellyfin, authentik, the lot. The services stay on their VMs; only the routing now lives in Git, with one wildcard certificate replacing per-route TLS fiddling. (One compatibility note: NPM never verified backend certificates, so traefik gets --serversTransport.insecureSkipVerify=true for parity — the self-signed Proxmox and UniFi backends object otherwise.)

Auditing the routes also turned over rocks. Four routes pointed at services that no longer existed. And the Cloudflare tunnel on my auth VM — surely important? Its published hostname had no public DNS record: a door connected to Cloudflare’s edge every day, reachable by no one, for a year. Deleted. Migration by removal is the most satisfying kind.

End state: NPM serves exactly one route — its own admin page — pending its funeral.

The scoreboard

Two days, on a platform that was four days old:

  • 🪦 one VM deleted (4 GiB + 22 GB reclaimed), one VM emptied to a single GPU-bound service
  • 📺 nine stateful services migrated with zero data loss and a one-command rollback that was never needed
  • 🔒 one VPN tunnel serving a whole namespace, with a proven kill-switch
  • 🌐 28 NPM routes → 1, the rest as code
  • 💀 two dead services, four dead routes and one ghost tunnel removed
  • 🤖 every image pinned and watched by Renovate

The thesis from part 2 held: DNS, TLS, secrets and monitoring absorbed all of it without a single new line of per-service configuration. What’s left for part 4 is the long tail — paperless off the NAS, authentik into the cluster, the jellyfin GPU question, and formally burying NPM — but the homelab’s center of gravity has already moved. The repo isn’t describing a corner of the lab anymore. It is the lab.