What is the difference between agent memory and RAG?

RAG, introduced by Lewis et al. in 2020, retrieves passages from an external document index to ground an answer, while agent memory is a store the agent itself writes to as it works, keeping experience, decisions and corrections for later turns and sessions. Anthropic describes agent memory as notes "persisted to memory outside of the context window" that "get pulled back into the context window at later times." In the CoALA framework (Sumers et al., 2023), RAG counts as retrieval from semantic memory, usually "a fixed, read-only semantic memory", whereas language agents "may also write new knowledge" into memory. The practical difference is the write path.

How does RAG work?

RAG pairs a language model with a retriever over a document index: the retriever finds the passages closest to the query, and the model generates an answer conditioned on them. In the original paper, Lewis et al. used "a dense vector index of Wikipedia, accessed with a pre-trained neural retriever", built from the December 2018 dump split into 100-word chunks, 21 million documents in total. The paper also showed that RAG knowledge is updated by replacing the index, not by retraining the model. From the model's side, RAG is read-only: people and ingestion pipelines decide what goes into the index and when it is rebuilt.

What does agent memory add that RAG does not?

Agent memory adds a write path and ranking signals beyond similarity. The agent records what happened, what it decided and what a person corrected, then reads it back on a later turn. The "Generative Agents" paper (Park et al., 2023) stores every observation in a "memory stream" with creation and last-access timestamps, and ranks memories by recency, importance and relevance, not by relevance alone. Those agents also write "reflections", higher-level conclusions stored with pointers to the memories they cite. Anthropic's memory tool for Claude follows the same idea: Claude creates, reads, updates and deletes files in a /memories directory that persist between sessions. A write path brings upkeep: notes go stale, and two notes can contradict each other. See what happens when memory conflicts.

How do RAG and agent memory compare?

The two differ mainly in who writes the store and how often it changes.

	RAG	Agent memory
Who writes it	People and ingestion pipelines	The agent during work, plus people
What it holds	Documents split into chunks	Episodes, learned facts, decisions, corrections
When it changes	When the index is rebuilt or replaced	Every turn or session
How items are ranked	Similarity to the query	Similarity plus recency, importance and scope
Typical failure	The answer is missing from the documents or outdated	Stale or conflicting entries pile up
Example question	"What does the refund policy say?"	"What did this customer agree to last week?"

Both can sit on the same storage. For that choice, see vector vs graph databases for agent memory.

Do you need both?

Most production agents need both, because they answer different questions. RAG alone is enough when the agent answers questions over a stable document set within one session and does not need to remember the user or earlier work. Memory becomes necessary when work spans sessions, when people correct the agent and expect the correction to stick, or when several agents continue each other's tasks. Anthropic's context engineering guide puts both under one goal: finding "the smallest possible set of high-signal tokens" for each step. Whichever you use, something has to decide what reaches the context window on each turn. See what context engineering is.

Where does elephant.broker fit?

elephant.broker, a pluggable runtime for AI agents, treats what your tools return and what past sessions left behind as inputs to one decision made on every turn: what your agent sees. The runtime assembles context from your tools, teammates and past sessions, ranks it, drops what is stale or off-task and places the working set in the window. After the turn, a post-turn hook captures decisions, corrections and outcomes and writes them back, so the next turn starts from them. elephant.broker does not replace your memory store: it uses a memory provider, Cognee by default, and other providers can be plugged in.

Sources

• Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020)
• Sumers, Yao, Narasimhan, Griffiths, Cognitive Architectures for Language Agents (2023)
• Park et al., Generative Agents: Interactive Simulacra of Human Behavior (2023)
• Anthropic, Effective context engineering for AI agents (Sep 29, 2025)
• Anthropic, Memory tool documentation
• elephant.broker: how a turn flows, features

Product details come from each vendor's public documentation as viewed on 2026-10-23 and may have changed. Product names are trademarks of their respective owners. Cognee is the default memory provider in elephant.broker; elephant.broker is not affiliated with the other products named.