Self-hosting Hindsight: agent memory on my Hermes box

Every agent I run has the same amnesia. Open a new session and it’s a blank slate — none of the preferences, none of the hard-won “don’t do that again” lessons, none of the context from last week. You end up re-explaining your own infrastructure to a thing that’s supposed to be helping you run it.

I’d actually tried to fix this once already. Months ago I installed Mnemosyne on my Hermes VM — a capable memory layer for agents — but I never properly wired it into my workflow, so it sat quietly accumulating a database I rarely read. That’s on me, not the tool; Mnemosyne is solid, and this is really me surveying the agent-memory space and trying out another option.

So this post is me finally doing it properly: standing up Hindsight (Vectorize’s agent-memory system) in Docker, carrying the useful memories over from my Mnemosyne store, and plugging the whole thing into my Hermes agents.

There were two genuinely interesting wrinkles along the way — the migration that isn’t a migration, and a rename that isn’t a rename. Let’s get into it.

What Hindsight actually is

Hindsight is built around three verbs:

• Retain — you hand it text; it extracts facts, identifies entities, and links them into a knowledge graph.
• Recall — four search strategies run in parallel (semantic, keyword, graph, temporal) to pull back relevant memories.
• Reflect — it uses recalled memories to generate a disposition-aware answer.

The bit I liked: the full Docker image does its own embeddings and reranking locally (a small BGE embedder + a MiniLM cross-encoder), so the only thing that needs an external LLM is the fact-extraction and reflection. I’m using GPT-5.5 for that part. That keeps the moving parts down and means most of the work never leaves the box.

flowchart TB
IN["📝 Retain<br/>raw text"] --> EX["🤖 LLM<br/>fact + entity extraction"]
EX --> PG[("🗄️ embedded<br/>Postgres + pgvector")]
subgraph local["🏠 all local in the container"]
EMB["🔢 BGE embeddings"]
RR["📊 MiniLM reranker"]
PG
end
Q["❓ Recall / Reflect"] --> EMB
EMB --> PG
PG --> RR
RR --> OUT["✅ ranked memories<br/>(+ reflected answer)"]

Where it runs, and the resource check

My Hermes VM (hermes, a single-NIC box on my huislab.app lab network) already has Docker, so that’s where it went. Quick sanity check before committing — the full image is chunky:

Plenty of headroom. One thing worth naming: Hindsight overlaps almost exactly with Mnemosyne, which already lives on this box. Mnemosyne’s a good tool — I’m not so much replacing it as evaluating the space — so I decided to run them side by side for a while rather than rip one out. More on the migration later.

The migration that isn’t a migration

Now for the old Mnemosyne data. I’d hoped for an export/import path. There isn’t one — they’re different engines with different data models, so nothing turnkey exists. What you can do is re-ingest: pull the text out of Mnemosyne’s SQLite store and Retain it into Hindsight, which rebuilds its own facts, entities and graph from scratch.

flowchart TB
DB[("🗄️ mnemosyne.db<br/>SQLite")] -->|read-only| EX["🔍 extract text"]
EX --> F{"filter for<br/>signal"}
F -->|"✅ 25 high-value<br/>facts/instructions"| R["📝 Retain into Hindsight"]
F -->|"❌ corrupted triples,<br/>truncated gists,<br/>raw chat turns"| BIN["🗑️ skip"]
R --> H[("🧠 Hindsight bank<br/>114 units / 120 entities")]

The reality check was humbling — though it says far more about my neglected setup than about Mnemosyne itself. Because I’d never tuned it or actually used it, the store had filled up with mostly low-signal data:

• consolidated_facts / facts — subject-predicate-object triples, but many in my store were fragmentary or truncated (objects cut down to things like "…is ctive" and "…is uto").
• gists — summaries cut off mid-sentence.
• 208 of 233 working_memory rows were raw [USER]/[ASSISTANT] chat turns I’d never curated.

The gold was a small seam: 25 high-value statements — Hermes recovery procedures, my delegation rules, timezone, the blog pipeline layout — with real importance scores. I extracted exactly those (read-only — the DB is live and Hermes still writes to it), tagged each with its original memory_type, and batch-Retained them into a fresh bank. Those 25 items fanned out into 114 memory units and 120 entities on Hindsight’s side, which is genuinely cleaner than what had accumulated in my untended store.

A quick Reflect query confirmed it stuck:

Q: What are the rules about delegating token-heavy work? A: Delegate all token-heavy tasks to Claude Code via persistent tmux sessions… don’t run them locally… don’t use claude -p print mode (per-call overhead)…

That’s my own rule, reconstructed from imported memory.

The rename that isn’t a rename

Small but instructive: I’d named the bank mnemosyne-import and then wanted it called hermes. Hindsight banks have a mutable display name but an immutable bank_id (it’s the URL slug). And export only serializes the bank template — config, directives, mental models — about 1 KB, not the memory units. So there’s no in-place way to change the id while keeping the data.

The only route is to re-Retain into a bank created with the new id, then delete the old one. Cheap enough at 25 items, but good to know before you assume export/import will carry your memories.

Wiring it into Hermes

The payoff. Hermes has a memory setup command that installs the client and asks for the API URL. Since Hermes and the Hindsight container live on the same box, the answer is just the default:

Hindsight API URL [http://localhost:8888]:

Press enter — the container publishes 8888 on the host, so localhost resolves straight to it. No LAN IP needed, and it keeps working even if the box’s address changes.

flowchart TB
subgraph vm["🖥️ Hermes VM"]
HERMES["🤖 Hermes agents<br/>(writer / researcher / publisher)"]
subgraph dock["🐳 Docker"]
HS["🧠 Hindsight<br/>:8888 API · :9999 UI"]
end
HERMES -->|"localhost:8888<br/>Retain / Recall / Reflect"| HS
end
ME["🧑 me"] -->|"hermes.huislab.app"| HERMES

Now the agents can write things down once and recall them next session — across restarts, across conversations.

The caveats worth knowing

It’s not free of strings:

• It still needs an external LLM for Retain and Reflect. Embeddings and reranking are local, but fact extraction and reflected answers go out to whichever LLM provider you configure.
• It’s a snapshot, not a sync. My Mnemosyne import was a one-time pull. If you want both kept in step you’d need an incremental re-ingest of newer rows.

Where this leaves me

Hindsight is running on the Hermes box, seeded with the handful of facts worth keeping from the old store, and reachable by my agents over localhost. The thing that finally made memory real for me wasn’t the install — that’s an afternoon — it was being honest about how little of the old data was actually worth carrying forward.

If you’ve got an agent setup that keeps forgetting its own hard-won context, this is a surprisingly low-friction way to give your bots a long-term memory.