Agent memory vs RAG vs context windows: three systems, three jobs

An agent can have a large context window, retrieve the correct handbook page, and still forget a user preference on the next run. The three capabilities are often described with the same word, memory, even though they solve different problems.

The practical distinction is simple: a context window carries the current working set, RAG retrieves inspectable source evidence, and agent memory preserves durable conclusions that should change later behavior.

The short answer

• Context windows answer: what information can the model see during this turn?
• RAG answers: which source passages are relevant to this question?
• Agent memory answers: what durable knowledge from earlier work should influence this decision?
• A production agent usually needs more than one layer, but each layer needs its own authority, scope, retention, and correction rules.

The comparison starts with authority

Storage technology is not the most important difference. A vector index can support both document retrieval and memory search. The real difference is what makes a record authoritative and what should happen when it changes.

Context window: temporary working state

The context window is the material assembled for one model call or agent run. It may contain system instructions, recent messages, tool results, retrieved memories, document excerpts, and an active plan.

Its normal lifecycle is short. The application should rebuild it for each task, enforce a token budget, and discard intermediate material that no longer affects the decision. A larger window lets the application send more text, but it does not decide what deserves to survive or which version is current.

RAG: source-backed evidence

Retrieval-augmented generation searches a corpus whose authority comes from preserved source material: manuals, policies, contracts, research, tickets, or product documentation. A useful hit should retain a path back to the document, version, and exact text that supported it.

Chunks, embeddings, summaries, and entity projections are retrieval aids. They should remain replaceable so a parser, chunking strategy, or embedding model can improve without losing the original evidence.

Agent memory: durable decision state

Agent memory stores smaller conclusions that should influence work across turns or runs. Examples include a stable preference, a project constraint, a verified outcome, a corrected account fact, or a troubleshooting path that already failed.

A memory needs structural ownership, stable identity, timestamps, history, and a way to update, supersede, expire, or delete it. Similarity search alone does not provide those guarantees.

One support request can use all three

Suppose a customer asks why an export still fails. The agent may assemble:

1. Context: the current question, the last two messages, and the latest tool response.
2. RAG evidence: the current export manual and the relevant workspace policy, with source identifiers.
3. Memory: a verified conclusion from an earlier ticket that this customer uses a custom timezone and already tried a browser restart.

The model sees one bounded prompt, but the application retains the boundaries. Current account data can override an old memory. A policy quotation can be cited without turning it into a profile fact. The recent tool response can disappear when the run ends.

A decision rule for every new piece of information

Before storing or retrieving a record, ask four questions:

1. Is it needed only for the current task? Keep it in working context.
2. Does its authority come from a document or system of record? Keep the source and retrieve it through RAG.
3. Is it a durable conclusion that should affect a later decision? Store it as scoped agent memory.
4. Could two categories apply? Keep both representations linked instead of collapsing them. A durable conclusion may cite the document or ticket that established it.

This rule prevents the common shortcut of sending every useful sentence into one vector collection and hoping retrieval will reconstruct the intended lifecycle.

Write loops are different

Context assembly

Context is compiled, not appended forever. Select recent turns, trusted tool state, relevant memory, and source excerpts under an explicit budget. Label each block so retrieved text is not mistaken for instructions.

Document ingestion

Retain the original file or exact text, assign a stable source identity, extract losslessly when possible, and build deterministic chunks and retrieval projections. Updating a source should create an inspectable new version or replace the source under a clear contract.

Memory writes

Write only when the application has observed something durable. If the input is a conversation that needs extraction, use a visible asynchronous inference operation. If the application already knows the exact conclusion, store it verbatim without another model call.

Recall loops are different too

A memory query should bind trusted user, agent, run, and project scope before ranking candidates. A document query should preserve source identity and neighboring evidence. Context assembly should then decide which results are safe and useful for the current request.

Do not ask the model to choose its own tenant boundary. Do not put source citations inside an unlabelled memory block. Do not assume the highest similarity score is automatically current or authoritative.

Budget the compiled prompt by role

All three layers eventually compete for the same model input budget. Treat that budget as an application contract rather than letting whichever retriever returns first fill the window.

• Instructions: reserve stable space for system policy and tool rules.
• Current request: keep enough recent dialogue and task state to resolve references and intent.
• Durable memory: include only the small set of preferences, constraints, and prior outcomes that can change this decision.
• Source evidence: prefer the passages needed to answer or cite the current question, with source identifiers intact.
• Tool state: include current results and failures, then discard them when they stop affecting the workflow.

When the budget is tight, compress repeated conversation state before removing a hard constraint or the source passage needed to justify the answer. Record which candidates were omitted so a retrieval failure is distinguishable from a prompt-budget decision.

Define a minimum contract for each durable layer

A memory record needs correction semantics

At minimum, keep stable identity, content, structural scope, provenance, creation and update time, validity when known, and change history. A search result should retain the memory ID so feedback or correction can target the record that influenced the answer.

A document result needs source semantics

Keep document identity, source key, version or content hash, exact retrieved text, and position within the source. Neighboring chunks can add context, but they should not replace the path back to the canonical document.

Working context needs assembly evidence

The context itself may be ephemeral, but the application should be able to explain which memory and source candidates were selected for an important request. Log identifiers and timing under an application request ID without copying secrets or complete private prompts into general logs.

The operational cost of choosing the wrong layer

Putting durable preferences only in chat history increases repeated tokens and makes deletion ambiguous. Putting manuals into memory loses provenance and makes re-indexing risky. Putting every inferred conclusion into source RAG makes correction and current-state selection an application-wide query problem.

The cost appears during incidents: operators cannot identify which record was authoritative, retries create duplicates, a deletion clears one index but not another, or a model receives both the old and corrected claim. Clear layer boundaries reduce that search space before any ranking improvement is attempted.

Four architectures that fail quietly

Using complete chat history as long-term memory

Cost grows with every turn, corrections remain mixed with obsolete claims, and relevant decisions become harder to locate. Conversation continuity is useful, but it is not a durable record model.

Using document chunks as user memory

A policy paragraph becomes detached from its version and retention rules. When the source changes, the application cannot tell whether the profile fact should change too.

Using inferred memory as source evidence

A compact conclusion may be useful for behavior, but it can omit qualifications. When an answer requires proof, retrieve the source rather than presenting an inference as a quotation.

Using context fields as authorization

A user ID inside a prompt is text, not an access-control decision. The server must derive project and structural scopes from trusted authentication before retrieval begins.

Where FishMem fits

FishMem separates canonical memory records from source-preserving Documents. Memory records carry structural scope, history, temporal fields, and explicit lifecycle operations. Documents retain exact source content while chunks and indexes remain rebuildable projections.

The application still owns the working context. It decides which memories and document excerpts enter an OpenAI, Anthropic, or other model run, how they are labelled, and which current system data can override them.

Frequently asked questions

Does a larger context window remove the need for RAG?

No. A larger window changes capacity, not source selection, provenance, versioning, or retention. RAG remains useful when the corpus is larger than the request budget or the answer needs inspectable evidence.

Does RAG remove the need for memory?

No. A source corpus is not automatically a current user profile or a history of verified workflow outcomes. Durable memory gives those conclusions a smaller, correctable lifecycle.

Can the same vector database support both?

Yes, but shared infrastructure does not imply a shared authority model. Keep record types, scopes, source links, deletion behavior, and correction paths explicit.

Should every model response become memory?

No. Store only conclusions that are durable, self-contained, scoped, and worth correcting later. Drafts, reasoning traces, acknowledgements, and raw tool output normally remain outside long-term memory.

Start with three boxes

Draw three boxes labelled working context, durable memory, and source corpus. For one real agent request, list every input and output in the correct box. Then define how retrieval compiles a bounded prompt without erasing those boundaries.

Read What is agent memory? for the full write and recall architecture, or Memory is not context for a deeper lifecycle treatment. The source-backed RAG cookbook shows how the two retrieval paths meet in application code.

You can inspect the open-source FishMem engine or create a free FishMem Cloud workspace and test one request across separate runs.