skip to content
BitsAndBytes
Table of Contents

Part 4 ended with a promise: paperless off the NAS, and the long-tail cleanup that follows. This is that post — and it turned out to be less about paperless than about a single sentence my brain kept circling back to:

Proxmox is where compute lives. The Synology is my data bucket.

That sentence sounds obvious. It is not. It quietly decides where your database goes, whether you stand up a whole new storage layer, and how you sleep at night when a single VM holds documents you can never re-download. Most of this migration was me arguing with that sentence until it told me exactly what to build.

The setup

Paperless-ngx is my document archive — every scanned invoice, contract and letter, OCR’d and searchable. It was running in Synology Container Manager as a five-container stack: the webserver, a PostgreSQL 16 database, a Redis broker, plus Gotenberg and Tika for document parsing. Roughly 210 documents. On disk: a whole 146 MB.

That tiny number matters, because it kept puncturing every grand plan I reached for.

The argument with myself: where does the data go?

The lazy migration is to drop everything onto local-path — k3s’s built-in storage class, which just carves directories out of the node’s own disk. Fast, simple, zero new moving parts. And it’s exactly what every other stateful app in my cluster already uses.

But local-path on a single-node cluster means my irreplaceable documents would live on one VM’s virtual disk. No RAID. No Hyper Backup. Gone if that disk dies. The Synology — sitting right there with redundancy and backups — is the obvious home. So: NFS-mount a Synology share and point paperless at it?

Mostly yes. But not for the database.

flowchart TB
APP["📑 paperless pods<br/>VM 108 · compute"]
APP --> PG["🐘 Postgres pgdata<br/>local-path on VM 108 · block, NFS-unsafe"]
APP --> M["📄 documents + index<br/>Synology NFS · plain files, safe"]
APP -. "document_exporter · 03:30 nightly" .-> B["💾 nightly export<br/>Synology · RAID + Hyper Backup"]

Postgres on NFS is a known foot-gun — file locking and fsync semantics over NFS invite corruption. So “just NFS everything” was wrong for the one component that holds all the metadata: which document is which, its tags, its correspondent, its date.

That left three honest options:

  1. Synology CSI / iSCSI — present a block-storage LUN from the NAS, install a CSI driver, and Postgres runs on NAS-backed block storage safely. The proper answer. It makes the Synology the data bucket for the whole cluster, not just paperless.
  2. All-NFS — including pgdata. Simplest, and quietly dangerous for the DB.
  3. Hybrid — documents on NFS, pgdata on local-path, and a nightly logical export to the NAS as the real source of truth.

I genuinely talked myself into option 1 for a while. Then the 146 MB punctured it again: standing up an entire iSCSI CSI layer — a new driver, a DSM service account, LUN management — to durably store 146 MB is not engineering, it’s cosplay. The CSI driver is the right call the day I want NAS-backed block storage as a general cluster capability. It is absurd as a vehicle for one tiny app.

So the principle resolved into something sharper than where I started. The rule isn’t “no bytes may touch Proxmox.” My cluster already keeps Prometheus, Grafana and a dozen app configs on local-path — and that’s fine, because those are regenerable. The real rule is narrower and truer:

Irreplaceable data belongs on the NAS. Regenerable state can live on compute.

Which lands me on the hybrid. Live data on local-path (fast, simple, consistent with the rest of the cluster), and the authoritative, restorable copy of the documents pushed nightly to the Synology — where RAID and Hyper Backup do their job. The live database can sit on Proxmox precisely because its real backup lives on the bucket. And capacity was never a constraint: the node had 41 GB free, sitting on a host pool with 678 GB to spare, growable with a single qm resize.

The migration that walked around a wall

My old notes warned of a blocker: paperless’s data lives under /volume2/docker/@docker, root-only, and I have no sudo on the Synology. Raw-copying a Postgres data directory out of there was going to be a fight.

Two realizations dissolved the fight entirely.

First: I already own most of the data. The compose file sets USERMAP_UID: 1031 — that’s my own Synology UID, set so I’d have write access to the consume folder. Which means the documents and media are owned by me, readable over plain SSH. The only thing genuinely behind the root wall is pgdata, owned by the Postgres container’s UID 999.

Second — and this is the one that mattered: don’t copy pgdata at all. Paperless ships a first-class migration tool, document_exporter, that runs inside the app, reads the database as the application, and serialises everything — documents, tags, correspondents, users, permissions — into a portable directory with a manifest. No raw database files. No UID-999 wall. And because it’s a logical export, it’s engine- and version-agnostic.

flowchart TB
E["📦 document_exporter<br/>(inside Synology container)"] --> S["🗂️ stage to /volume1/data/paperless-import<br/>(already NFS-mounted by the cluster)"]
S --> J["⚙️ one-off import Job in k3s<br/>document_importer"]
J --> DB["🐘 cluster Postgres"]
J --> MED["📄 cluster media volume"]

The slick part is the middle step. The export needs to get from the NAS into the cluster — normally a dance of scp then kubectl cp. But the cluster already NFS-mounts /volume1/data (my media library lives there). So I staged the export into a folder on that exact share, and the cluster could read it directly. No copy out, no copy in. The export was simply already there.

On the Synology, over SSH:

Terminal window
sudo docker exec -u root paperless-webserver-1 \
  bash -c 'document_exporter /usr/src/paperless/export --no-progress-bar \
           && chown -R 1031:65536 /usr/src/paperless/export'
cp -a /volume2/docker/paperless-ngx/export/. /volume1/data/paperless-import/

Then a one-off Kubernetes Job ran document_importer against the empty cluster instance. The result, first try at the data layer: 210 documents, 22 correspondents, 20 tags, 1,900 database objects — including my user account and permissions, so I logged into the new instance with my existing password. The whole time, the Synology instance kept running, read-only and untouched. Rollback was always one docker start away.

The same trap, three times

Here’s the bit that’s almost funny in retrospect. The first import Job died instantly:

s6-envdir: fatal: unable to envdir /run/s6/container_environment: No such file or directory

The paperless image’s convenience commands — document_exporter, document_importer — aren’t the real programs. They’re thin wrappers that call s6-envdir /run/s6/container_environment … to load the environment that the container’s s6 init sets up at boot. When you override a container’s command (as a Job does), s6 never runs, that directory never exists, and the wrapper face-plants before doing anything.

The fix is to skip the wrapper and call the actual Django management command, with the env supplied directly by the pod:

workingDir: /usr/src/paperless/src
command:
  - /bin/bash
  - -c
  - python3 manage.py document_importer /source/paperless-import --no-progress-bar

Then it tripped on a Django system check — PAPERLESS_CONSUMPTION_DIR is not writeable — because I hadn’t mounted the consume volume into the import Job. Mount it, and the import sailed through.

And then — because I apparently don’t learn — the exact same s6 trap was lurking in my nightly backup CronJob, which I’d written days earlier using the same wrapper. I caught it only because I refused to trust a backup I hadn’t watched run. I triggered it manually:

Job has reached the specified backoff limit — Failed

Same s6-envdir death. Same fix: python3 manage.py document_exporter, plus mounting the consume volume so the system check passes. A backup you have never seen succeed is not a backup — it’s a hope with a cron schedule. Manually firing it before trusting it turned a silent 03:30 failure into a thirty-second fix.

The transferable lesson: container images that wrap their tools in an init system are great for docker exec into a running container, and a landmine the moment you run them as a fresh pod with an overridden command. Reach past the wrapper to the program underneath.

The secret that had been dead for eighteen hours

The best bug of the migration wasn’t even mine.

When Flux first deployed the stack, the database and webserver pods sat in CreateContainerConfigError. The cause: my ExternalSecret — the thing that pulls the database password and secret key from 1Password — was failing with:

could not get secret data from provider
wasm error: out of bounds memory access

The 1Password provider runs the vendor’s SDK as a WebAssembly module inside the external-secrets controller, and its WASM heap had wedged. But here’s the unsettling part. I checked the other ExternalSecrets in the cluster — Cloudflare, Grafana, AdGuard, every one — and all of them were failing too, and had been for eighteen hours.

Nothing had visibly broken, because external-secrets keeps the last-good Kubernetes Secret around when a sync fails. Every running app still had its credentials cached. The rot was completely invisible — right up until paperless, the one brand-new secret in the cluster, needed its very first sync and found a corpse.

flowchart TB
W["🧩 1Password WASM SDK<br/>heap wedged ~18h ago"] --> F["❌ every ExternalSecret sync fails"]
F --> H["🫥 but existing Secrets are cached<br/>→ running apps notice nothing"]
H --> N["🆕 paperless needs a FIRST sync<br/>→ no cache to fall back on"]
N --> V["👀 the silent outage finally becomes visible"]
V --> R["♻️ restart the controller<br/>→ fresh WASM heap → all 6 recover instantly"]

The fix was almost insultingly small — kubectl rollout restart deployment external-secrets for a fresh WASM heap, then nudge a re-sync — and within seconds all six secrets went green. But the lesson is the one worth keeping: a credential cache is a wonderful way to hide an outage from yourself. A whole subsystem had been down for the better part of a day, and the only reason I found out is that I happened to deploy something new. That’s now a monitoring alert on my list: ExternalSecret not-ready, regardless of whether anything has visibly broken yet.

The cutover

With real data verified in the cluster, flipping the live hostname was the easy part — and a clean demonstration of why part 2’s groundwork keeps paying off. paperless.huislab.app already resolved to the cluster (the *.huislab.app wildcard from part 4 points everything at traefik). It had merely been routed back out to the Synology via an external-route Service. Cutting over meant:

  • add a wildcard-flat Certificate to the paperless namespace (cert-manager issued the Let’s Encrypt cert in thirteen seconds),
  • point the namespace’s own Ingress at the in-cluster webserver,
  • delete the old external route and let Flux prune it.

No DNS change. No client reconfiguration. The hostname simply stopped going to the NAS and started going to the pod, with a valid TLS cert, the moment Flux reconciled. One curl confirmed it: HTTP 200, Let’s Encrypt cert, resolving to the cluster.

The scoreboard

One evening, on a platform six days old:

  • 🗄️ a storage principle sharpened into a rule — irreplaceable data on the NAS, regenerable state on compute — and an iSCSI layer correctly not built for a 146 MB app
  • 📦 210 documents migrated with the official logical exporter, zero raw database files touched, the source instance never stopped
  • 🧵 one NFS share doing double duty so the export needed no copy in or out
  • 🪤 the same s6-wrapper trap found and fixed in two places — the second only because I refused to trust an unwatched backup
  • 🕵️ an eighteen-hour silent secrets outage uncovered, fixed in one command, and turned into a monitoring gap to close
  • 🔁 a hostname cut from NAS to cluster with no DNS change and a 13-second TLS cert
  • 💾 nightly logical backups landing on the bucket, verified running before being trusted

Paperless was supposed to be a routine migration, and in the mechanical sense it was. The value was in the arguing: a vague instinct — “the Synology is my data bucket” — forced into a precise rule that will decide every stateful migration after this one. The platform absorbed the move, as it has the last three. The thinking was the work.

Next on the long tail: authentik into the cluster, the jellyfin GPU question, and the funeral for Nginx Proxy Manager I keep promising and keep postponing.