By Rudy · Partner & Head of Data and AI Platforms
“Pippin is still running. Mostly behaving.” I ended the last article on that line and promised to come back to memory. It’s been exactly six months. For an article about memory, that’s either the worst possible look or the most honest opening I could ask for. Somehow it’s both.
Pippin, for anyone who doesn’t know him, is my agent: a cat, and the sheriff of the town.
We had OpenClaw and Hermes Agent. This fall it’s Claude Duet, OpenAI Dots, Grok Bot, Meta’s Muse – the same shape, new names, for people whose weekends belong to cars and gardens rather than config files. None of this took a prophet to predict; it just took a year to ship. The gap between what a hobbyist builds and what a company can release to everyone is mostly lag.
When does an agent actually need memory?
While the lag ran its course, I was running Pippin. And Pippin isn’t a coding agent. I’ve run coding agents for almost two years, and I still can’t tell you for sure they need memory in the way people mean it. The output is only *sometimes* better with it. A good workflow produces its own memory – requirements, specs, plans – and quality artifacts are more reliable than any memory file, because a spec that gets superseded decays gracefully instead of hanging around to contradict the new one.
A long-running assistant is a different animal. The better part of a year into running Pippin for my household, what he needed to remember was birthdays, where things live, gift ideas, who talks to whom about what. A flat file grows until you can’t find anything in it, the same fact duplicated to hell. The assistant needs memory far more than the coding agent ever did – and that difference is where this whole story goes sideways.
What I tried, and where it fell short
It started, as these things do, with a couple of files. `history.md`, append-only in theory. `memory.md`, edited in place by a tool call or a scheduled job. It persisted, it grepped, and for a while that was enough.
This worked fine until it didn’t. Pippin kept corrupting the formatting in `history.md`, so I did the reasonable thing and added cleanup rules – a tidy-up pass to fix the drift. And the cleanup pass hallucinated, and rewrote the entire history. Gone. Replaced with something confidently false, built from whatever we’d last talked about. (Yes, I know about backups and version control. This was avoidable.)
That one stuck with me, because it’s not really a memory lesson – it’s a tooling lesson. You can’t let a model freely self-edit something as load-bearing as a person’s memory with nothing but good intentions and a prompt. You need surgical, well-defined tools for that. It’s part of why I moved to Hermes Agent around that point: surgical editing instead of freeform rewriting, and a real searchable session database instead of a flat `history.md` that one bad pass can nuke.
As some of you will relate, I tried to get my better half into this as well. And eventually the household stopped being a one-person problem: me and my partner, Basia, one big shared pile. The flat-file approach didn’t just creak, it broke outright.
Then I started seeing all the cool projects out there – so many different approaches – and I had to try them all. Every project turned out to be a whole opinion about memory – what to extract, how to store it, who it’s for. Half of them weren’t even solving my problem: some are for customer support, some for profiling, some for teams with an org chart. And some just don’t work.
I tried a whole pile of them: Mem0, Mnemosyne, Hy Memory, Honcho, Holographic, OpenViking, ByteRover, and a few more whose names I’ve already forgotten.
Mem0, the popular one. Thirteen problems logged in two weeks – back then; I haven’t been back since. Most were a mismatch: it’s built for consumer apps, not a long-running agent. Two stuck with me. About 2,000 of my 2,800 memories turned out to be Pippin’s own output. And a stale assumption kept out-scoring its own correction, forever. A memory that can’t correct itself gets more confidently wrong the longer it runs.
Mnemosyne, back then, was a lovable mess. It kept adding new things without stabilizing the old ones, the documentation was a hallucination, and my instance was soon full of junk. It needed more hands than I could spare, and I had more projects to try. It has come a long way since.
MemPalace I never actually tried – I just enjoyed the memes. Milla Jovovich. Yes, the actress from The Fifth Element and Resident Evil. She built an AI memory system that went viral on a “perfect” LongMemEval score. Then the fine print came out: cooked benchmarks. I’ve always been skeptical of benchmarks – benchmaxxing has been around as long as benchmarks have.
Hy Memory – one day I was bored, or desperate, or both, and I followed a random Reddit post about it. It kinda worked, but half the features in the code were never actually wired up. Making it work as advertised meant hot-wiring it myself, because it wasn’t open source back then. It got open-sourced just this week.
Honcho and Holographic – nothing much was wrong with either, so no war stories here. Honcho’s actually exciting, and one day I’ll give it proper treatment. Holographic memory is a forty-year-old idea that kinda actually works.
OpenViking – already talking about being a company brain, a knowledge base for teams. I got in between major releases. A lot had changed, and not everything just worked; some of the ideas I’d picked it for were gone. I also tried it for work, with a coding agent, and that didn’t go well either – I could only afford the cheapest models for it. The tokens were spoken for. But it left me with the question that shaped everything after: who owns the store, the agent or the human?
A couple of these deserve their own footnote. I tried the context-tree ones – ByteRover, and OpenViking’s – on cheap models to keep the cost down. The information scattered: duplicated, inconsistent, everywhere. And when I tried to force structure back onto it, I made it worse, not better. I’m honestly not sure what the lesson is – whether context trees are just fragile, or whether I under-paid the models for a job that needed a sharper one. Possibly both. I’m not going to pretend otherwise.
Landing on Hindsight
To be fair to the ones that didn’t make the cut: I’m fairly sure I could have made OpenViking – or something like TencentDB Agent Memory – work, given enough effort. Hindsight didn’t win some technical bake-off. It just worked for me, mostly, with less effort than the alternatives wanted. And the one thing it got right that none of the others did was the one that mattered most to me: shared versus private, in a two-person household.
Hindsight was the one I kept coming back to. Three times now. The first time, it worked well for a week, and then I moved on to the next thing on my list – I swear it was a phase. The second time, I ran a massive shared bank for a few months, and the cost hit me – not in the memory system, in my agent’s context: every recall dragged so much of the bank into the conversation that retrieval was costing twice the agent’s own turn. The third time, I did the three-bank thing.
Three banks: one for me, one for Basia, one shared. Two birds, one stone – it fixed the retrieval cost and the ownership problem at once. I could have kept one shared bank and separated things with tags. But a bank is a wall, and a tag is a convention. Nothing crosses over unless someone deliberately puts it there.
And it had to be self-hostable. Hindsight’s vector store is nothing exotic – Postgres and some extensions on top – which is exactly why it self-hosts comfortably. For personal and household data, self-hosted isn’t optional for me.
Inside the memory bank
Inside the bank, extraction comes first. The conversation gets captured every few turns, or on hooks – a new session, a compaction, the end of one. What gets pulled out is governed by a mission: which facts to keep, who they belong to, and just as important, what not to store.
Alongside the facts, extraction pulls out entities – as tags, as metadata. The schema isn’t fixed; it adapts to the bank. My personal bank sorts by life domain – work, home, family. The shared bank sorts by who a fact is actually about.
This attribution part matters more than it sounds. I once had Pippin pull a friend’s LinkedIn profile while we were sorting out some staffing. The session got ingested whole, and the extractor read it chunk by chunk – the chunk holding one job title never met the chunk holding my friend’s name. Default attribution did the rest. By the next morning my memory held a career I never had: someone else’s entire work history filed under my name, sitting on top of my own. Not a few wrong facts – a whole life, replaced.
Extracted content lands first as world facts – append-only. They don’t get edited, only added to. Observations sit a layer above that: consolidated, and they actually change. One can overrule another, or be marked true only for a certain period.
Why this matters came clear through a fact that started as nothing: a minor, well-managed allergy, medicated and under control. It got registered once, then kept getting pulled up every time we planned something. And it echoed – a little louder each pass, its own echo chamber – until three months later Pippin was planning around someone who apparently needed full-time care. A true fact, repeated until it became a false one.
Recall is cheap to run. Zero LLM calls – the cost of memory is paid when you write it, not when you read it. But cheap to run is not the same as cheap to carry: read results still ride into your agent’s context, which is exactly how the big shared bank bled me. Reading runs four searches in parallel: semantic, keyword, graph, temporal. And if you want the paid, hallucinated version, there’s always reflect – recall with an agent doing tool calls and reasoning on top, and the answer written by the model.
The four searches produce four kinds of numbers that can’t be compared – cosine similarity, tf-idf, graph hop weights, temporal decay. RRF fuses them by rank, not by score, so you never have to normalize anything. That’s the whole trick. And the reranker on top isn’t an LLM either. The default is a local cross-encoder, which scores each candidate against the query one pair at a time. I swapped it for Jev, a System One model: a tiny model that ranks instead of writes. It sees all the candidates at once, as options in a single multiple-choice question, and the answer is the ranking.
The takeaway
I don’t trust the memory machine completely – the allergy echo and the overwritten-history mess made sure of that. So next to it sits a separate knowledge base: my Obsidian vault, the ground truth and the shared workspace between me and Pippin. When memory looks suspicious, Pippin checks the vault. Some may ask why not just use the vault as the memory system. The answer is scale and retrieval cost. Two tools, two jobs.
So what would I tell someone starting out? Memory is case-by-case. You might not need one at all, and bolting one on anyway can do more harm than good. It’s not a ranking – different use cases, different tools. Mem0 for a multi-user personal trainer app. OpenViking for a team on a shared product. Hindsight for a two-person household that needs shared and private kept apart.
And here’s the closing irony I can’t resist: my best-performing coding agent uses no memory system at all – it just self-edits a handful of markdown files, by rule. Sometimes the right shape for your use case is no memory system whatsoever.
I forgot a lot, building toward all of this – including, apparently, to write it down. Pippin didn’t forget, though.
Have thoughts on memory for your own agents, or a failure story of your own? Reach out to Rudy on LinkedIn – he collects these.
