Teaching my agent what to forget

Yesterday I stood up Hindsight — Vectorize’s agent-memory system — on my Hermes box, so my agents would stop forgetting everything between sessions. The pitch for this whole category of tooling is seductive: your agent gets smarter every day. Auto-retain, auto-reflect, auto-managed memory. Set it up, walk away, watch it learn.

I didn’t walk away. A new system earns that kind of trust; it doesn’t get it on day one — and “auto-everything, writing to a permanent store” is exactly the sort of default I want to understand before it’s been quietly running for a month. So instead of turning it loose, I spent a day reading how people actually tune these things.

That reading is what stopped me. There’s a failure mode that people running this exact stack keep describing — and once you see how the pieces are wired, you realise it isn’t bad luck, it’s baked in. Left on its defaults, this kind of memory doesn’t quietly get smarter. It gets confidently wrong, and the architecture makes it worse over time. The culprit isn’t the memory engine. It’s who gets to write to it.

The promise, and the trap underneath it

Here’s the default setup with most of these systems: after each turn, the agent writes a summary of what just happened into long-term memory. Helpful, right? Except the agent is also the most biased narrator in the room. It just finished a task, it thinks it did great, and that’s what gets written down — not what actually happened, but the agent’s flattering summary of it.

Next session, it recalls that summary as a settled fact. The fact confirms its prior belief. It gets a little more confident. It writes that down too. Repeat.

flowchart TB
A["🤖 Agent finishes a turn<br/>'nailed it'"] --> B["💾 Writes its own<br/>summary into memory"]
B --> C["🗄️ Stored as a 'fact'"]
C --> D["🔁 Next session:<br/>recalled as settled truth"]
D --> E["💪 Belief reinforced —<br/>confidence ratchets up"]
E --> A
style E fill:#b91c1c,color:#fff
style A fill:#1d4ed8,color:#fff

This is a closed loop with no ground truth in it. A wrong belief, recalled and re-confirmed a few times, becomes harder to correct than a wrong belief stored once — because now there’s a pile of self-generated “evidence” behind it. Frequency starts masquerading as proof.

You don’t have to run it for a month to see this coming — you can read it straight off the wiring. And the thread I’d been reading was full of people who had run it that long, describing exactly this from the other side. The consensus fix was blunt and unanimous: turn auto-retain off. But “off” on its own just means your memory stops growing. The interesting question is what you replace it with.

The fix in one sentence

Separate the writer from the reader.

The agent should be allowed to read memory all day long — that’s recall, that’s the useful part. But it should never be the thing that writes memory. Writing should be done by a separate process that doesn’t have an ego in the game, working from the actual transcript of what happened — not the agent’s opinion of it.

flowchart TB
subgraph reader["🤖 The agent — READ ONLY on memory"]
R["Recall: pulls facts<br/>to inform its answers"]
end
subgraph writer["✍️ External writer — the ONLY thing that writes"]
T["Reads the real transcript"]
F["Filters it through a<br/>strict 'what matters' policy"]
D["Derives durable facts"]
end
MEM[("🗄️ Memory bank")]
MEM -- recall --> R
T --> F --> D -- retain --> MEM
style reader fill:#1e3a8a,color:#fff
style writer fill:#065f46,color:#fff

Once you see it this way, every symptom from that thread lines up. The confidence loop exists only because the writer and the reader are the same entity. Cut them apart and the loop can’t close — the agent’s self-congratulation never becomes a memory, because the agent isn’t holding the pen anymore.

So I rebuilt my setup around that one idea. Three pieces.

Piece 1: make the agent a reader

The first move is the easy one. In Hindsight’s per-bank config there’s an auto_retain flag, and the Hermes side has a knob for how often it flushes. I turned auto-retain off. Recall (auto_recall) stays on, because that’s the half I actually want.

That’s it for the agent. From here on it can look things up, but it cannot write its own story into the record.

Piece 2: teach the writer what’s worth keeping

A writer that records everything is almost as useless as a biased one — you just drown the signal in transcript noise. Hindsight lets you attach a retain mission to a bank: a plain-language policy that runs server-side and decides what actually becomes a memory versus what gets discarded.

This is where you encode judgement. Mine, trimmed down:

EXTRACT — discrete facts, each anchored to an entity (host, service, project, person, plan):
  • Decisions and their rationale — the "why", in the user's own words
  • Stable preferences, conventions, standing constraints
  • Resolved outcomes with a before→after — not in-progress states
  • State changes that REMOVE a prior fact: a service decommissioned, a plan cancelled

IGNORE:
  • Transient command output, one-off debugging chatter, greetings
  • The assistant's own claims and self-assessment — never store "I succeeded" as a fact
  • Secret values of any kind — record that a credential exists, never the value

PRINCIPLES:
  • Provenance first: prefer what the user stated over what the assistant asserted
  • Frequency is not evidence; repetition does not make a thing true
  • On conflict, the user's most recent correction supersedes the old fact
  • When in doubt, discard. A missing fact costs less than a polluted memory.

The two lines doing the heavy lifting against the confidence loop are “never store the assistant’s self-assessment as a fact” and “the user’s most recent correction supersedes the old fact.” That’s the antidote written down explicitly.

One filter I didn’t expect to need: live state versus setup. My agents touch Home Assistant constantly, and HA is a firehose of readings — “it’s 21°C in the bedroom”, “the door is unlocked”. None of that is a durable fact; it’s stale the instant it’s stored. The rule that saved me: retain the wiring, never the reading. “The morning automation dims the lights at sunset” is a fact worth keeping. “The lights are dimmed right now” is telemetry, and telemetry has no business in long-term memory.

Piece 3: the writer itself — and the trap I walked into

With the agent muzzled, something has to actually do the writing. The clean version is a small script that reads the real conversation transcript out of the agent’s own session database, runs the slice through the retain mission, and stores the result. The agent never calls it. It’s wired to fire on lifecycle events.

My first instinct was: flush on session end. Tidy. And immediately, a sharp question stopped me cold — what if I leave a session open? I chat with one of my agents in the same thread for hours. If the only trigger is “session closed”, then a whole day of conversation sits unwritten, and if the box reboots before the session ever closes, it’s just gone.

So a session-end-only writer is a trap. You need a second trigger that fires during long sessions, plus a backstop in case a trigger ever misfires. Three triggers, one shared idempotency check:

flowchart TB
CONV["💬 Conversation happening"] --> T{"Trigger fires"}
T -->|"session ends / resets"| B["Boundary flush"]
T -->|"every 8 user turns"| P["Periodic flush"]
T -->|"daily cron (safety net)"| C["Sweep-all flush"]
B --> WM
P --> WM
C --> WM
WM{"New messages since<br/>last watermark?"}
WM -->|"no"| NOOP["✅ do nothing"]
WM -->|"yes"| SLICE["Read new slice →<br/>filter → retain"]
SLICE --> ADV["Advance watermark"]
style SLICE fill:#065f46,color:#fff
style NOOP fill:#374151,color:#fff

The watermark is what makes this safe. It’s just a per-session bookmark: the id of the last message already written to memory. Every flush only looks at messages past the watermark, then moves it forward. That means the three triggers can never step on each other — the periodic flush and the boundary flush and the cron all share the same bookmark, so nothing gets written twice. A flush with nothing new to do is a clean no-op.

That daily cron at the bottom is the piece that lets me sleep. It re-checks every session and sweeps up anything the live triggers somehow missed. Because it exists, turning auto-retain off is no longer scary — even if every hook silently broke, the cron would still catch everything within a day. The backstop is what makes the whole “agent is read-only” decision safe to commit to.

Does it actually work?

Honest answer: it’s early, and I’m watching it rather than declaring victory. But the structural problem is gone in a way I can reason about. The agent physically cannot write its own press releases into memory anymore. What lands in the bank now comes from the transcript, filtered by a policy I can read and edit, and a correction I make today outranks whatever it might otherwise have talked itself into.

The thing I keep coming back to is how little of this was about the memory engine. Hindsight was fine. Mnemosyne would’ve been fine. The bug was architectural, and it was the same bug in every “your agent gets smarter every day” product I looked at: they let the student grade their own exams and file the results as transcript.

Memory you can trust doesn’t come from asking the agent what it learned. It comes from the record of what actually happened — and from being ruthless about what you let in.

If you’re running this stack and want the gory details — the exact config knobs, the writer script, the hook wiring — give me a shout and I’ll do a follow-up with the full setup.