I Built AI Agent Memory Wrong 3 Times. Here's What Finally Worked.

I've been building memory systems for AI agents for the past two months. I read every paper I could find — A-MEM, SimpleMem, HiGMem, Memory OS, Mem0, MemMachine. I tried vector databases, knowledge graphs, multi-layer hierarchies, and five-stage extraction pipelines.

I got it wrong three times before I got it right. Here's what the experience taught me.

Why Memory Matters More Than We Think

The AI industry talks a lot about reasoning, tool use, and planning. Memory gets a footnote. But memory is what transforms a chatbot into something that actually feels like it knows you.

If a user comes back after six months, your agent should greet them with relevant context — not a blank stare. That's a simple bar, and almost nothing clears it today.

Three Attempts, Three Lessons

Attempt 1: The Kitchen Sink. I built five parallel subsystems — a vector search engine over markdown files, a SQLite entity graph, flat profile files, a trajectory store, and an observation engine. Five stores, five formats, five retrieval paths. The read path ran five parallel lookups and mashed the results together.

It worked, technically. But there was no single source of truth, the context window bloated with redundant information, and debugging anything meant tracing through five different systems. I had ~2,500 lines of memory code and couldn't confidently explain what the agent “knew” at any given moment.

Attempt 2: The Academic Pipeline. Inspired by the literature, I built a five-stage write pipeline: compress, extract, classify (ADD/UPDATE/DELETE/NOOP), consolidate, index. Three LLM calls per incoming message. A three-layer hierarchical store — turns, events, and knowledge. Heat formulas with exponential decay. Memory evolution with retroactive updates.

Beautiful architecture. This is where the worst bug happened.

The Bug That Changed My Approach

My agent had a friend stored in memory: “Joe — close friend, works in Applied AI at Anthropic.” Then, in a conversation about job searching, I mentioned sending my resume to Joe for feedback.

The extraction pipeline saw “resume” + “Anthropic” + “Joe” and inferred a role that was never stated. The UPDATE mechanism confidently overwrote “close friend, Applied AI” with “Anthropic recruiter.” Every subsequent interaction compounded the error.

A week later, I asked the agent to help me write a message to Joe. It drafted a cold, transactional follow-up: “He's a busy Anthropic recruiter. That line is filler. Just say what you want.”

About my close friend.

One bad inference, permanently committed to memory. That's when I realized the extraction pipeline wasn't just complex — it was dangerous.

Attempt 3: Simple and Correct. I threw away the 2,500 lines and replaced them with ~450. The key insight wasn't technical — it was philosophical: stop trying to be clever at write time. Store what was actually said. Retrieve it smartly. Let users correct you when you're wrong.

What the Research Actually Showed Me

I went through seven major papers on agent memory. Three findings reoriented my thinking:

Retrieval improvements outperform ingestion improvements 12:1. MemMachine (April 2026) ran an ablation study that surprised me. Retrieval-stage optimizations gave +9.4% cumulative improvement. Ingestion-stage optimizations gave +0.8%.

I had spent 80% of my effort on the write path — extraction, classification, consolidation — and treated retrieval as basically solved. The data says to flip that ratio entirely.

ADD-only outperforms CRUD. This was the finding that validated my gut feeling after the Joe bug. Mem0 v2 had a sophisticated memory lifecycle where every new piece of information was classified as ADD, UPDATE, DELETE, or NOOP. The system would find related memories, resolve contradictions, and merge duplicates.

Mem0 v3 dropped UPDATE and DELETE entirely. Just ADD. Their benchmark score jumped from 71.4 to 91.6. Removing the ability to modify memories made the system dramatically better. The reason tracks with my experience: write-time conflict resolution is exactly where errors compound.

Simple systems have a higher ceiling than expected. ChatGPT's memory for 200 million users is roughly 33 plain-text sentences injected into every conversation. No vector database. No RAG pipeline. No knowledge graph. The “Poor Man's Memory” study (March 2026) found that a control agent with zero memory scored within 0.02-0.03 points of agents with sophisticated memory systems on task-completion benchmarks.

A Framework for What's Worth Remembering

The hardest question in memory isn't technical. It's editorial: should I store this at all?

I developed a heuristic I call the Alex Test. Imagine Alex, a senior analyst who's been in a role for years. Alex doesn't write down everything — that would be drowning in notes. But Alex has learned what matters: the preferences that change recommendations, the context that shifts interpretation, the history that prevents repeating old mistakes.

Before storing a memory, ask: would Alex note this down for a colleague who's taking over?

• Durability — Will this still matter in 30 days? “User prefers weekly reports on Monday” passes. “User is currently in a meeting” doesn't.
• Decision Impact — Will this change a future response? If it won't affect agent behavior, it's not worth the context window space.
• Explicitly Stated — Was this said directly, or are you inferring it? This is the critical filter. “User sent resume to Joe” is observation. “Joe is a recruiter” is inference. A skilled human analyst wouldn't assume someone's job role from a single data point.
• When in Doubt — Store with low confidence rather than discard. If the signal repeats, confidence grows naturally. If it never comes up again, recency decay handles it. Hold hunches loosely.

Scoring Retrieval

When retrieving memories, I rank them with a composite score:

score = 0.50 × semantic_similarity
     + 0.25 × recency
     + 0.15 × confidence
     + 0.10 × log(access_count + 1)

Recency uses exponential decay with a 30-day half-life. A memory from today scores 1.0, from 30 days ago scores 0.5, from 90 days scores 0.125.

The confidence hierarchy by source is what makes this work in practice:

User explicitly said "remember this"  → 1.0
User corrected the agent             → 0.9
Same fact repeated across sessions   → 0.8
Agent chose to store it              → 0.7
Background extraction inferred it    → 0.4

With this hierarchy, a background-extracted “Joe is a recruiter” at 0.4 confidence can never outrank a user-corrected “Joe is my friend” at 0.9. The scoring formula becomes your immune system against the exact class of bugs that burned me.

What I'd Build Today

Storage: Markdown files or a simple SQLite database. Human-readable, auditable, easy to debug. One index always loaded, topic files loaded on demand. You can add a vector index later if keyword search proves insufficient — but you'd be surprised how far keyword search goes.

Write path: Let the agent write its own memories inline during conversation. No separate extraction pipeline. The agent has the best judgment about what matters in the moment. For implicit knowledge, you can add background extraction later, scored at low confidence.

Read path: Auto-inject key facts at session start. Retrieve additional memories on demand — not every message needs the full memory context. This is where to invest your engineering time.

Maintenance: ADD-only, no automated updates or deletes. Users can correct or delete manually. Timestamps on everything. Recency scoring handles staleness naturally.

What I Would Not Build

And I say this having built all of them:

• A knowledge graph for fewer than 10K entities
• A multi-layer hierarchical store
• Write-time classification pipelines
• Complex heat formulas with exponential decay
• Memory evolution with retroactive updates

Each of these added complexity without proportional value. Your mileage may vary, but I'd start without them and add only when you hit a clear ceiling.

Memory Is a Product Decision

Here's what took me three attempts to internalize: what the agent remembers, how it retrieves, when it forgets — these aren't engineering decisions. They're product decisions. They shape how users experience your agent.

A user who feels understood comes back. A user who has to repeat themselves churns. Memory matters not because it moves benchmark scores (the research suggests it barely does), but because it builds trust.

Start simple. Store text. Retrieve smart. Let users correct you. Everything else is optimization you can add when you need it.