Context management in agents:

How to build a memory

Every agent you use

remembers you now

Everyone agrees how to build it.

We tried it, but shipped something else.

Alyx: the agent we built memory for

Our users jump around,

and memory has to follow them

Why not just keep

the whole conversation?

• Sessions end
• Sessions rot
• Sessions get compacted

Part 1:

How everyone says to build memory

The industry agrees:

memory is a search problem

• IBM · Mem0 · Redis · AWS · Zep · every explainer blog

Step 1: an agent with a context window

Step 2: extract memories in the background

Step 3: three kinds of memory, three of everything

Step 4: a knowledge graph that understands time

Step 5: hybrid search, then pick a reranker

Step 6: forgetting, safety, locks, isolation

And when it's tuned, it works

• Zep:
• Up to +18.5% accuracy on LongMemEval
• Context: ~115,000 tokens → ~1,600

Part 2:

We built it

Our first prototype

was built on Graphiti

Problem 1: every write is a pile of LLM calls

~10 LLM calls per turn.

20+ with custom types.

Problem 2: broad preferences

never get retrieved

• "Make a dataset of agent-prod traces"
• vs.
• "This user wants datasets with exactly 70 rows"

Problem 3: memory becomes

a data science project

Part 3:

What actually ships

Full context beats

every memory system

• Full conversation in context
• beat every memory system on accuracy
• - Mem0's own paper

Never underestimate the power

of a command line tool

• Letta:
• GPT-4o mini + plain files + grep-style tools
• 74.0% on LoCoMo

ChatGPT just includes everything

• "OpenAI just includes everything with every message"
• - Shlok Khemani

Coding agents just use a file.

• CLAUDE.md · AGENTS.md
• Loaded in full, every session
• "Keep it under 200 lines"

The whole architecture collapses into one file

So that's what we shipped

• One file per user, per space
• 8,000 characters ≈ 2,000 tokens
• Always loaded
• No embeddings. Anywhere.

RAG is not dead

Why a file works

• Nothing to miss
• Cheap
• Easy to debug

Part 4:

Simple is not the same as easy

The complexity moves from reading memory to writing it

The bug changes too

• Graph: the right fact exists, but search misses it
• File: the right fact is in context, but isn't applied

Lesson 1: Don't make the agent decide mid-task

What it saved was junk

• trace IDs · encoded asset IDs · filter strings · counts

Only remember what's durable

and can't be fetched

• Durable - still true next month
• Not re-fetchable - no tool can reconstruct it
• Useful across sessions - shapes many future requests

Lesson 2: Write memory in the background

The turn summarizer is not optional

Lesson 3: Edit the file, don't rewrite it

• add → insert text after an exact anchor
• replace → swap one exact string for another
• replace with "" → delete

Even surgical edits lose things

Facts the user told you directly

get protected

• [user] datasets should have exactly 70 rows

Lesson 4: Compaction is the hard part

"No, I can't make it any shorter"

• Compaction ran on 22 of 48 turns
• Never once reached its target
• 3 attempts: 9,009 → 9,005 characters

LLMs can't count,

so give them percentages

• Tell it the current length
• Tell it the target length
• Tell it how much shorter, as a percentage
• Ask for 6,000 → reliably get under 8,000

Lesson 5:

Memory can make your agent worse

Test memory in both directions

• Memory-dependent: needs memory to answer well
• Memory-corruptible: memory could make it worse

Multi-session scenarios,

through the real agent

16 of 18 when memory matters.

9 of 9 when it shouldn't.

• Writes: 44 of 48 turns correct
• Memory-dependent: 16 of 18
• Memory-corruptible: 9 of 9

Memory is untrusted input

Part 5:

When a file isn't enough

One file works

because Alyx's scope is narrow

Always-loaded core,

search for the long tail

Add complexity

when an eval tells you to

Thanks

• arize.com/blog/alyx-agent-long-term-memory-architecture

Follow me:

LinkedIn: Laurie Voss

BlueSky: @seldo.com 🦋