GitOps homelab part 4: the DNS that ate itself

Part 3 ended on a victory lap: Nginx Proxy Manager down to one route, the media stack behind a VPN, every route in Git, and a tidy claim that DNS, TLS, secrets and monitoring now absorb new work for free. Two days later the homelab handed me a more honest sentence:

the ads came back.

Not because a blocklist broke. Because the thing doing the blocking had no real backup — and the “backup” it did have was quietly worse than nothing. Part 4 is about giving DNS a second pair of legs, and the afternoon the cluster nearly ate its own.

The symptom: a backup that un-blocks your ads

DHCP was handing every device two DNS servers: AdGuard on guardian (192.168.1.155) and the gateway (192.168.1.1). The trap is that clients don’t treat those as primary and fallback — they treat them as peers, and on the slightest blip they drift to the second one and stick there.

And the gateway resolves the internet just fine — the unfiltered internet, with no idea what proxmox.huislab.app is. So “redundancy” actually meant “a resolver that silently turns ad-blocking off and breaks every internal name the moment a device wanders onto it.” The disease wasn’t the ads; it was that DNS had exactly one good resolver, and its understudy was a downgrade.

The real fix is therefore not “remove the gateway.” It’s “build a second good resolver so you can remove the gateway.” That’s the whole arc of this post.

A second resolver, not a second house of cards

AdGuard already runs on guardian. The lazy move is to drag it into the cluster — but DNS on a single-node k3s cluster means a house-wide outage on every node reboot, plus a bootstrap circularity (the thing that resolves your container images is the thing you’re trying to deploy). So the primary stays exactly where it is. Instead I added a replica in k3s and let adguardhome-sync mirror the primary’s entire config onto it — blocklists, custom rules, rewrites, upstreams. Edit once on the primary; the replica follows within minutes.

flowchart TB
C["💻 LAN client<br/>DHCP gives it DNS1 + DNS2"]
C --> P["🛡️ AdGuard primary<br/>guardian · 192.168.1.155"]
C --> R["🛡️ AdGuard replica<br/>k3s · 192.168.1.240"]
P -. "adguardhome-sync mirrors<br/>blocklists · rules · rewrites · upstream" .-> R
P --> U["🧠 Unbound<br/>recursive · DNSSEC"]
R --> U

Crucially the replica is not a dumb forwarder. Because the sync copies the upstream setting too — guardian’s Unbound — the replica blocks ads and knows internal names. A genuine peer, not a degraded one. Exactly the thing the gateway never was.

The DNS that ate itself

To expose the replica at 192.168.1.240:53 I reached for a Kubernetes LoadBalancer. On k3s that’s klipper (ServiceLB), which implements the service’s host port with a CNI firewall rule. And that rule matched every packet headed to a local address on port 53 — with no destination-IP filter at all. Which includes the node’s own loopback resolver at 127.0.0.53.

So the node’s own DNS queries got redirected into the AdGuard service. Which had no pods yet — the image was still pulling. Which couldn’t pull — because the node’s DNS was now broken. Read that again as a loop, because that’s exactly what it was:

flowchart TB
A["📦 LoadBalancer exposes the replica on :53"] --> B["🔀 klipper installs a DNAT for :53<br/>matching every local address — no filter"]
B --> C["💀 the node's own queries to 127.0.0.53:53<br/>get hijacked into the AdGuard service"]
C --> D["⛔ that service has no pods —<br/>its image is still pulling"]
D --> E["🔁 the pull needs DNS…<br/>which is now broken"]
E --> C
C -. "the escape" .-> F["✅ point the node at 192.168.1.155 directly<br/>a remote IP isn't a 'local' destination → dodges the DNAT"]

The cruel part was the misdirection. resolvectl query resolved everything perfectly the whole time — it talks to systemd-resolved over D-Bus, bypassing port 53 — so the resolver looked healthy while getent, dig, and containerd all silently failed. I burned a good ten minutes trusting the one tool that was lying to me.

The escape is a one-liner of insight: a remote IP isn’t a local destination. Point the node’s resolver straight at the primary (192.168.1.155) instead of its own loopback stub, and the queries sail right past the DNAT. A netplan override plus repointing /etc/resolv.conf, and the deadlock dissolved. Bonus: the node also stops drifting to the flaky gateway — the same disease as the clients, cured on the node for free.

The lesson worth carrying: exposing a privileged, infrastructure-critical port like 53 through a node’s own service load-balancer is a foot-gun. The node has to be moved off that port’s local path first, or it strangles itself.

Two smaller bruises

The replica then OOMKilled itself. I’d given it a 256 MiB limit; the instant the sync applied the real blocklists — and AdGuard holds every rule in RAM — it blew past the ceiling and CrashLooped with exit code 137 and, infuriatingly, no error in the log. An OOM kill leaves no parting message; you only find it in the pod’s Last State. Raised to 1 GiB; it now idles around 190 MiB.

Collapsing the phonebook

With the replica healthy, the deeper flaw surfaced. Both AdGuards forwarded *.huislab.app lookups to one Unbound on guardian, which held ~25 hand-edited split-horizon entries (sonarr → .240, grafana → .240, and so on, one per service). Guardian dies and the replica still blocks ads and resolves the public internet — but every internal name goes dark.

The fix leaned on the sync I’d just built. Move the split-horizon out of Unbound and into a single AdGuard DNS rewrite: *.huislab.app → 192.168.1.240 (traefik, the universal front door since part 3). Because the sync replicates rewrites, both resolvers now answer internal names themselves, before they ever consult Unbound. Twenty-five hand-edited lines became one — and new traefik routes need zero DNS work, because the wildcard already covers any name I’ll ever add.

flowchart TB
subgraph before["❌ before — ~25 hand-edited Unbound lines on guardian"]
direction TB
B1["sonarr.huislab.app → .240"]
B2["grafana.huislab.app → .240"]
B3["…twenty-odd more, one per name…"]
B1 ~~~ B2 ~~~ B3
end
subgraph after["✅ after — one synced AdGuard rewrite"]
direction TB
A1["*.huislab.app → 192.168.1.240"]
A2["blog.huislab.app → Cloudflare (exception)"]
A1 ~~~ A2
end
before ~~~ after

The blog exception

A wildcard quietly assumes everything under your domain is internal. It isn’t. blog.huislab.app — this very page — is a Cloudflare-hosted Worker and must keep resolving to Cloudflare’s edge, never to traefik. AdGuard has no “resolve this one normally” passthrough, but an exact-match rewrite beats a wildcard, so blog gets a specific entry pointing at Cloudflare’s anycast IPs. (Safe to hardcode: Cloudflare routes by SNI, so any edge IP serves the right site.) It’s the single name on the exception list — and a standing reminder that “I own the whole domain” and “everything on it is internal” are not the same statement. Any future public name needs the same carve-out.

Unbound stays — and here’s why

Tempting question: if AdGuard now owns the phonebook, can Unbound be deleted? No, and the reason is the point. Unbound was doing two jobs — the split-horizon (now moved) and recursive resolution: walking the DNS tree from the root servers down, with DNSSEC validation, instead of handing every query you make to a big public forwarder. That second job is the entire reason to run it. So Unbound stays as AdGuard’s recursive upstream; it just sheds the phonebook role and goes back to what it’s genuinely good at. I trimmed its config from 78 lines to 37 — the dead per-name entries gone, recursion and a couple of real exceptions kept.

Failover that’s finally real

The last step was the one I’d been parking for months: change DHCP to hand out .155 + .240 and drop the gateway entirely. Now the two servers any client can land on are both full, ad-blocking, split-horizon-aware resolvers. There is no longer a bad resolver to drift to — the disease is removed, not medicated.

The proof: force a client to renew its lease, confirm it gets the new pair with the gateway gone, then check each resolver on its own still answers internal names and blocks ads. Both did. Guardian can now fall over completely and the house keeps resolving — ads blocked, internal names intact, public internet up via the replica’s fallback. That’s the redundancy the gateway pretended to be.

The scoreboard

One afternoon, on a platform a week old:

• 🛡️ a second AdGuard resolver living in k3s, its config auto-synced from the primary
• 🕳️ one self-inflicted DNS deadlock — a LoadBalancer that ate the node’s own resolver — diagnosed and escaped
• 📒 ~25 hand-edited split-horizon entries collapsed into one synced wildcard rewrite (plus exactly one exception)
• 🧠 Unbound trimmed to a pure recursive, DNSSEC-validating upstream
• 🔁 DHCP failover that survives the primary dying — forced-renewed and verified on a live client
• 🚫 the original “ads came back” gone structurally, not patched

Part 3’s thesis was that the platform absorbs new services for free. Part 4 is the platform absorbing its own resilience — and a reminder that the most embarrassing outages are the ones you build yourself, with a LoadBalancer and a port number. Next on the list: paperless off the NAS, and the long-promised funeral for Nginx Proxy Manager.