skip to content
BitsAndBytes
Table of Contents

Part 1 ended with a closed loop: Flux reconciling a Git repo, SOPS-encrypted secrets, Renovate opening update PRs. Also: a cluster that nothing and nobody could actually reach. Impressive plumbing, zero taps.

Part 2 is where it becomes visible — names in the browser, padlocks in the address bar, dashboards with graphs. But the real theme is something I only saw clearly at the end: every layer in this post is work you do once, and then never again per service. The last section is the proof.

flowchart TB
DNS["🧭 one DNS rule<br/>*.k3s.huislab.app"] --> ING["🌐 traefik ingress<br/>one routing rule per app"]
ING --> TLS["🔒 one wildcard cert<br/>auto-renewing, default for all"]
TLS --> ESO["🔑 secrets pulled from 1Password<br/>Git holds references only"]
ESO --> MON["📈 first real tenant<br/>arrives with zero extra setup"]

One DNS rule for the whole cluster

My LAN already runs split-horizon DNS: AdGuard Home in front for filtering, Unbound behind it holding the zone logic. Inside the house, *.huislab.app resolves to my Nginx Proxy Manager VM; the internet sees something else entirely. The cluster gets a sub-zone carved out of that:

local-zone: "k3s.huislab.app." redirect
local-data: "k3s.huislab.app. IN A 192.168.1.240"

Two lines in Unbound. redirect means the zone and everything under it answers with that one address — traefik’s LoadBalancer IP. And because Unbound picks the most specific matching zone, whoami.k3s.huislab.app goes to the cluster while jellyfin.huislab.app keeps going to NPM. The old proxy and the new one coexist for as long as the migration takes; nothing flips until I say so.

flowchart TB
B["💻 browser asks:<br/>whoami.k3s.huislab.app?"] --> AG["🛡️ AdGuard<br/>(filtering)"]
AG --> UB["🧭 Unbound<br/>(zone logic)"]
UB -- "*.k3s.huislab.app" --> T["☸️ traefik<br/>192.168.1.240"]
UB -- "*.huislab.app" --> NPM["🦑 nginx proxy manager<br/>192.168.1.155 (legacy)"]

The first consumer is whoami, a debug service that echoes back the request it received. Its routing rule is an Ingress — host-header routing as four lines of YAML in the repo, next to the deployment it routes to. This is the part that quietly changes the economics: NPM routes live in a database I click at; Ingresses live in Git, get reviewed, get rolled back, and adding service number twenty involves no DNS work at all. The wildcard already covers it.

TLS on a cluster the internet can’t see

Next: the padlock. cert-manager automates Let’s Encrypt, but the usual HTTP-01 challenge (“prove you control this domain by serving a file on it”) is impossible here — Let’s Encrypt can’t reach a LAN-only cluster. The DNS-01 challenge can: prove control by writing a TXT record into public DNS via the Cloudflare API. It’s also the only challenge type that issues wildcards, which is what I wanted anyway: one *.k3s.huislab.app cert for everything.

One subtlety that cost me a real think: my existing NPM wildcard is *.huislab.app, and a wildcard only covers one label level — it does not match whoami.k3s.huislab.app. The sub-zone needs its own cert. Fine: that’s one more Certificate resource in Git, renewing itself forever.

flowchart TB
C["📜 Certificate<br/>*.k3s.huislab.app"] --> O["📋 Order<br/>(ACME paperwork)"]
O --> CH["🎯 Challenge<br/>DNS-01"]
CH -- "Cloudflare API token" --> TXT["☁️ TXT record<br/>_acme-challenge..."]
TXT -- "Let's Encrypt verifies" --> CERT["🔒 signed cert<br/>→ k8s Secret"]
CERT --> TS["🗄️ traefik TLSStore 'default'<br/>every Ingress gets it"]

Getting there produced three war stories worth writing down:

  1. The trailing newline. I piped the Cloudflare token out of 1Password with op read — which appends a newline. Cloudflare’s error message deserves credit for literally asking: “does the API token contain a newline?” It did. op read -n exists for exactly this.
  2. Split-horizon bites back. Before cert-manager asks Let’s Encrypt to verify, it checks DNS propagation itself — using the cluster’s resolver, which is my LAN resolver, which claims to own huislab.app and knows nothing about Cloudflare. The self-check could never succeed. The fix is telling cert-manager to do its checking against public resolvers: --dns01-recursive-nameservers=1.1.1.1:53 plus --dns01-recursive-nameservers-only. If you run split-horizon DNS, you will hit this.
  3. The backoff trap. After fixing the token I sat watching nothing happen — ACME failures back off exponentially, and the fix had landed half a minute after a retry. Deleting the stuck Challenge resources is safe; the Order recreates them immediately and you skip the wait.

The closing move is the quiet one: traefik’s default TLSStore points at the wildcard secret, so every Ingress — current and future — gets TLS without mentioning TLS, and a global HTTP→HTTPS redirect makes plain-text impossible. Certificates are now nobody’s job, including mine.

External Secrets: Git stops holding secrets at all

Part 1 ended with an honest footnote: the SOPS encrypt-then-commit dance is human-error-prone, and the endgame was the External Secrets Operator with 1Password behind it. That’s now real, and the mental flip is the direction of travel:

flowchart TB
subgraph before["before: secrets pushed into Git (encrypted)"]
direction TB
ME1["🧑‍💻 me"] -- "read → encrypt → commit" --> GIT1["📦 repo<br/>(ciphertext)"]
GIT1 --> CL1["☸️ cluster"]
end
subgraph after["after: cluster pulls from the vault"]
direction TB
GIT2["📦 repo<br/>(references only)"] --> CL2["☸️ ESO in cluster"]
CL2 -- "read-only service account" --> OP["🔐 1Password vault"]
OP -- "synced hourly" --> SEC["☸️ k8s Secret"]
end
before ~~~ after

ESO’s modern 1Password provider talks to the vault directly with a service account token — no sidecar Connect server to host. The Kubernetes side is an ExternalSecret: a small, plaintext-safe YAML naming an item and field in the vault and the Secret it should become. Rotate the token in 1Password and the cluster picks it up within the hour. No re-encrypting, no commit, no human.

There’s one honest catch — the bootstrap paradox. ESO needs a credential to reach 1Password, and that one credential cannot come from 1Password. So it lives as the single remaining SOPS-encrypted file in the repo. SOPS didn’t retire; it got demoted to one job.

The proof was migrating the Cloudflare token: delete the encrypted file, add a 16-line ExternalSecret, then force a fresh certificate issuance through the synced credential. Issued. The pipeline is now 1Password → ESO → Secret → cert-manager → Cloudflare → Let’s Encrypt, and Git touches none of it.

Belt and braces: a gitleaks pre-commit hook (in a versioned .githooks/ directory, activated with git config core.hooksPath .githooks) refuses any commit containing secret-shaped strings. I tested it by committing a fake AWS key. Blocked. Good hook.

The first real tenant — and the payoff

Time to make the platform earn its keep: kube-prometheus-stack, the standard monitoring bundle — Prometheus, Grafana, Alertmanager, node-exporter, kube-state-metrics and the Prometheus Operator, one umbrella chart.

The chart assumes a “normal” cluster, and k3s isn’t one, so three adjustments mattered:

  • Disable the etcd / scheduler / controller-manager / kube-proxy scrape targets. k3s compiles them into one binary; the separate pods the chart expects don’t exist, and you’d stare at four permanent false TargetDown alerts.
  • Give Prometheus a real volume. The default is emptyDir — your entire metrics history evaporates on every pod restart. Twenty gigs of local-path PVC and 14 days of retention fixed that.
  • serviceMonitorSelectorNilUsesHelmValues: false — the worst-named flag in the ecosystem, and the difference between Prometheus scraping only its own chart versus discovering any app in any namespace that publishes metrics. Traefik, Flux and cert-manager all already speak Prometheus; they’re two-line YAMLs away from appearing in Grafana.

And here is where the compounding showed up. Grafana needed a URL, a TLS cert, and an admin password. The bill:

  • DNS: nothing — the sub-zone wildcard already covers grafana.k3s.huislab.app
  • TLS: nothing — the default TLSStore hands it the wildcard cert
  • Password: one Login item in 1Password plus an ExternalSecret referencing it

One git push. Eighty-one seconds later: six components running, 13 of 13 scrape targets up, and https://grafana.k3s.huislab.app serving dashboards behind a Let’s Encrypt padlock. Every layer built earlier absorbed the new tenant automatically — and Renovate is already watching the chart for updates, so the monitoring stack maintains itself the same way everything else does.

Where this lands

The platform phase is done. What exists now, all of it declared in one private repo:

  • 🧭 a DNS sub-zone where any new name just works
  • 🌐 ingress routing as reviewed, revertable YAML
  • 🔒 wildcard TLS nobody renews by hand
  • 🔑 secrets pulled from 1Password, with Git holding only references and one bootstrap file
  • 📈 monitoring as the first tenant that proved the platform absorbs new services for free

Part 3 is the migration grind: moving the docker-compose stacks in one by one, retiring NPM route by route, and wiring ServiceMonitors so the things that run the house show up on the dashboards. The foundation stopped being the story; now it’s just the floor.