Agents That Actually Learn: Beyond Session-by-Session Amnesia
We rebuilt the memory system three times before it stopped making things up. Here is what the third version actually does — and how we know it is working.
The first version of our memory system retrieved things confidently and often incorrectly. It would surface a "decision" from session 12 that no one had actually made, stated in authoritative language that passed without scrutiny because it sounded like something we would have decided. We discovered this only because the agent started acting on those fabricated memories — making choices inconsistent with the actual project history.
We threw it out. The second version was more conservative but not measurably better. Objects were stored as raw text, retrieved by keyword proximity, and returned without any indication of confidence, source, or age. The system would surface a lesson from 40 sessions ago as if it were equally relevant as something from yesterday.
The third version is what we are running now. It is more complex, slower to build, and meaningfully better. This is what we learned.
The Architecture: Typed Objects, Not Raw Text
The central design decision: every piece of memory is a typed object, not a text blob. At the end of each session, a single LLM pass extracts structured objects across seven types:
Each type has a different schema, different retrieval weight, and different lifecycle rules. A decision object carries the choice, the rationale as an array of items, the rejected alternatives with rejection reasons, and a version chain that links it to later decisions that supersede it. A lesson carries the generalized pattern, the sessions that produced it, and a confidence score. A capsule is a session summary — fast, high-level, used to orient the agent at the start of new work.
The retrieval layer is hybrid: FTS for keyword matches, vector similarity for semantic matches, and a query router that decides which layers to activate based on query intent. A query like "what permission model did we choose?" routes to the decision axis. A query like "what happened with the Bulwark meeting?" routes to capsules and events. A query like "how do we handle selector failures?" routes to lessons and procedures.
The Bugs That Taught Us the Most
Three bugs in this system were more educational than any design decision.
The version bug. We upgraded the extraction prompt from version 3 to version 5 with meaningful changes — better grounding rules, specificity gates, EvidenceItem schema. But the write path was still hardcoding extractor_version: 3 in two INSERT statements instead of using the constant. Every object extracted with the new prompt was being written as version 3, invisible to the v5 query path. The system appeared to be running v5 while silently accumulating v3 objects. We ran multiple sessions before discovering this, which meant a week of extractions had been attributed to the wrong version. A single wrong integer in two SQL statements.
The subject normalization bug. Subject keys were generated by LLM and not normalized. The same person would generate subjects like rahul-chinthala-rahul-chinthala, rahul-chinthala, and user-rahul depending on how the session was phrased. These were being stored as separate version chains, meaning facts about the same person were fragmented across dozens of disconnected subjects. We wrote a three-strategy normalization function — exact-half dedup, prefix-repeat dedup, entity-aware dedup — and ran a migration that normalized 3,941 rows across 2,417 subjects.
The read path bug. This one was the most expensive. Employee-mode channels in Discord were suppressing context injection — memory was writing successfully every session, but the read path gate (an isEmployee check) was silently preventing the built context from being used. The Telegram bot had the same problem in the other direction: it had a write path but no read path at all, meaning every Telegram conversation had access to nothing from the memory store. We discovered this only when patterns from Telegram conversations started failing to show up in memory-dependent queries — which required constructing a test that explicitly checked cross-surface recall.
The common thread: the failure modes of memory systems are silent. The system does not error. It just quietly fails to build the context it is supposed to build, and you only notice when something downstream behaves incorrectly.
How We Measure Whether It Is Working
After the first version's fabrication problem, we became serious about measurement. The metric we use is retrieval completeness on a fixed gold query set.
The gold set is 39 queries across five strata — questions that should retrieve specific information: decisions with their rationale, lessons with their evidence, facts about specific systems, procedures, events. Each query has a ground-truth object or set of objects that a correct retrieval should surface. An LLM judge (Claude Haiku, label-blinded) scores each retrieval as present, partial, or absent.
Completeness started at 28% when we first ran the gold set. As of this writing it is at 49%. The improvement came primarily from two places: fixing the suppression gate in the query router (which was routing entity-primary queries away from the decision axis entirely), and fixing the judge prompt to be role-aware (the judge was scoring alternative_considered elements as current-state facts instead of as rejected alternatives, which caused it to systematically undercount the decision stratum).
We maintain three numbers because they measure different failure modes. Completeness measures whether useful information is surfaced. Fabrication measures whether invented information is surfaced. Flat-fact stability measures whether changes to the extraction pipeline are silently degrading coverage of straightforward factual queries. You can improve completeness while introducing fabrication. You can fix fabrication while regressing flat-facts. The three numbers need to move together.
The Wrong Diagnosis
We spent three sessions building graph traversal infrastructure — a system that would walk semantic edges between memory objects and assemble decisions with their rationale and rejected alternatives as a unit. The hypothesis: the 60% completeness gap was a retrieval assembly problem. Connected information existed in the store but needed to be joined at query time.
We built it. We measured it. It was mechanically correct and produced zero improvement in completeness. The graph had 0.26 edges per object on average. Most decision objects had zero outgoing edges. There was nothing to traverse.
The actual problem — which we identified only after the graph traversal arc came back null — was that the query router had a suppression gate called primaryAxisSatisfied that was preventing 24 of the 39 missing elements from ever entering the candidate pool. They were present in the store. FTS found them. But the router was routing queries away from the axes that contained them.
The diagnostic that would have caught this took one session to run. The graph traversal arc took three. We could have done the diagnostic first. We did not, because the graph traversal hypothesis sounded more interesting.
The lesson: when a memory system is not surfacing information, check whether the information exists in the store before building retrieval machinery to assemble it. The retrieval gap is almost always a routing problem before it is an assembly problem.
What "Learning" Actually Means at the System Level
We are not fine-tuning model weights. The agent that operates today has the same underlying model as the agent from six months ago. What has changed is the structured store it has access to — and the quality of what gets into that store.
This is learning at the system level, not the model level. And it has real measurable effects: the agent's browser automation skills improve because stable selectors are cached and fallback hierarchies are extended. Decisions do not get re-made because the decision objects and their rationale are retrievable. Patterns of failure surface as lessons that modify future behavior.
The open problem is cross-context generalization. A lesson learned in one context — how to handle a particular type of DOM failure in LinkedIn — does not automatically transfer to a similar failure in a different tool. The extraction system does not yet identify abstract failure patterns across concrete instances. That is the next frontier.
If you are building agents that need to compound knowledge across sessions — and hitting the same walls we did — we would like to compare notes. rahul@ollie-labs.com