TrueMemory Shows Agent Memory Is Mostly a Retrieval Problem

We read the code and every benchmark file behind the local-first agent memory system. LoCoMo shows a real edge; the headline LongMemEval number is from the easy split.

SnackOnAI Engineering | Senior AI Systems Researcher | Technical Deep Dive | October 28, 2026

The promise: by the end of this issue you will know how TrueMemory captures and retrieves memories for coding agents, which of its published scores reflect real gains over simple baselines, and what its benchmarks do not test.

The fence: we cover the TrueMemory repository (latest commit on main August 29, 2026; PyPI 0.7.6.2), its README, the benchmark scripts and result files in benchmarks/, the Storage Is Not Memory paper, and the LongMemEval and LoCoMo benchmarks. We did not rerun the benchmarks; they need Modal plus paid OpenRouter calls. Our numbers come from the project's own result files and source code.

TrueMemory, from Sauron Labs, gives coding agents persistent memory across sessions. It installs lifecycle hooks into Claude Code, Cursor, Codex CLI and Gemini CLI, captures what you said, and injects relevant memories at the start of the next session. Everything is stored in one SQLite file, ~/.truememory/memories.db.

The research thesis, from the companion paper: storage is not memory. Systems that extract and discard information at write time lose facts that a later question needs, so TrueMemory keeps raw events and does the hard work at query time.

Caption: Two different write paths feed one retrieval engine. The benchmarks load raw messages straight into the database; your daily sessions go through the LLM extractor and encoding gate.

Four design decisions do the work.

One: SQLite is the whole stack. Keyword search (FTS5), vectors (sqlite-vec) and metadata live in one file. No server, no vector database, nothing to operate.

Two: fuse, don't choose. Keyword and vector results merge with Reciprocal Rank Fusion, which scores each memory by its rank in each list. A memory found by both retrievers almost always outranks one found by either alone.

Three: layers that degrade gracefully. Temporal filtering, contradiction timelines, a salience guard and a reranker each wrap in try/except, so a failing layer drops out instead of failing the search.

Four: tiers trade models, not architecture. All three tiers run the same pipeline. Edge swaps in an 8M parameter embedder and 22M parameter reranker that run on any CPU; Pro adds HyDE, which asks an LLM to draft a hypothetical answer and searches with that.

First, the fusion step, verbatim from truememory/hybrid.py.

Caption: RRF ignores BM25 and cosine score scales entirely. That is why a keyword index and a tiny embedding model combine well without tuning.

Second, the encoding gate, from truememory/ingest/encoding_gate.py, which decides what the auto-capture path keeps.

Caption: The module's docstring states raw ≥ 0.30; the normalization makes the effective raw threshold 0.225, and categories like corrections lower it further. Small, but it shows how tunable "what the agent remembers" is.

Third, our read of the project's own result files.

Caption: The LoCoMo gain is large and statistically solid. The LongMemEval gain over a plain vector database is smaller than TrueMemory's own run to run variation.

Input: in a Claude Code session you say, "We use FastAPI with Pydantic v2, and auth middleware lives in src/auth/." You close the session.

Step one, capture. The SessionEnd hook spawns background ingestion. The transcript is split into chunks of up to 20,000 characters (at most 20 chunks) and sent to an LLM extractor: local Ollama if it is running, otherwise the claude CLI, otherwise an OpenRouter, Anthropic or Groq key. It returns up to 50 atomic facts.

Step two, gate. Each fact is scored. An illustrative new fact with novelty 0.6, salience 0.4 and prediction error 0.1 gives raw 0.26, normalized 0.35: stored. A repeat of something already known, novelty 0.1, salience 0.3, prediction error 0.05, gives 0.13: skipped. Corrections ("we moved auth to src/security/") always pass.

Step three, store. Survivors are deduplicated and written to SQLite with an FTS5 entry and a 256 dimension vector.

Step four, recall. Next morning, SessionStart searches for context and injects the top memories as additional context. Your agent opens already knowing the stack.

Output: no repeated "what framework are we using?" questions. On the benchmark harness, retrieval plus answering averaged 1.01 s per question on Edge and 2.23 s on Pro, most of it the answer model's API call.

Caption: Edge, with 30M parameters of models, sits 3.4 points behind Pro. The pipeline, not model size, carries most of the gain over BM25.

Why it works. Conversational memory questions are mostly retrieval questions: find the two or three turns that hold the answer, then let the answer model reason. Keyword search catches names, file paths and numbers; vectors catch paraphrase; RRF merges them without score tuning; a cross-encoder reorders the top 100. Resolving relative dates ("last Tuesday") to absolute ones at ingestion, a preprocessing step in the LoCoMo config, helps temporal questions. And because it all lives in SQLite, there is no service to run.

What it trades away.

Insight One: the headline LongMemEval score is from the easy split, and there the gain over plain RAG is noise. The README advertises 92.0% on LongMemEval. The project's own benchmark notes say that is the oracle variant, where each question comes with only the relevant sessions, and that the paper reports the strict variant: 87.8%. On strict, plain ChromaDB RAG scores 87.0%, a 0.8 point gap with a one sigma uncertainty of about 2 points and a three run spread of 2 points. The README comparison table also omits RAG, BM25 and EverMemOS (94.5% on the same LoCoMo harness) while listing figures for MemOS and ReadAgent from outside it, despite the results doc noting its lenient rubric makes scores "not directly comparable to published LoCoMo baselines." TrueMemory's real, well supported win is LoCoMo: +6.8 points over RAG and half of RAG's errors removed.

Insight Two: the benchmarks measure a different system than the one you install. Every benchmark script calls engine.ingest() on raw conversation messages, the retrieval-first design the paper argues for. The hooks you actually install run the other path: an LLM extracts at most 50 facts per transcript, a weighted gate decides what to keep, and duplicates are dropped. No published number scores that path. If storage is not memory, the auto-capture pipeline's write-time filtering deserves its own benchmark.

On LoCoMo, BM25 alone scores 80.5% and TrueMemory Edge, with an 8M parameter embedder and a 22M parameter reranker on CPU, scores 89.6%, just 3.4 points below the Pro tier at less than half the per-question time. For agent memory, careful retrieval engineering over a single SQLite file buys most of the accuracy; bigger models and LLM query expansion buy the last few points.

Every time you open a new chat with an AI coding assistant, it has forgotten everything: your project, your preferences, the decision you made yesterday. You end up repeating yourself, like working with a brilliant colleague who has amnesia every morning.

TrueMemory gives the assistant a notebook. When a session ends, it writes down the important facts. When the next session begins, it looks up what is relevant and hands it to the assistant before you say anything. The notebook lives in one file on your own computer.

The clever part is not the notebook; it is the librarian. Finding the right two sentences among thousands of old messages is hard. TrueMemory searches by exact words and by meaning at the same time, then asks a small ranking model which results truly answer the question.

How well does it work? On one well known memory test it clearly beats simpler approaches. On another, it does about as well as a basic search tool. That tells us something about this whole field: much of what we call "AI memory" is really good search, and measuring it carefully matters as much as building it.

TrueMemory gets the architecture right: one SQLite file, hybrid search, rank fusion, a reranker, and hooks that make memory automatic. Its LoCoMo results show that careful retrieval beats both naive RAG and LLM-extraction memory systems like Mem0 by wide margins. But its marketing outruns its data. The headline LongMemEval number uses the easy split, where it ties plain RAG, and the auto-capture path you install is the one part no benchmark measures. Use TrueMemory for what it is: a well engineered local retrieval layer. Before trusting any memory system's leaderboard, ask which split, which baseline, and which ingestion path.

Your AI is only as good as what you feed it. Feed it a transcript with your product names, acronyms, and numbers spelled wrong, and you spend the afternoon hand-fixing the output.

Wispr Flow Notetaker uses your dictionary and calendar, so product names, acronyms, numbers, and uncommon names come out spelled right. It pops up when your meeting starts and captures Zoom, Google Meet, Teams, and Slack huddles with one click, no bot joining the call.

Then your meetings show up inside Claude or ChatGPT through the built-in connector. No copy paste. Try Notetaker free on Mac and Windows.

Try Notetaker Free