Memory is not context: a practical architecture for agent state

August 12, 2026Engineering12 min read

A decision framework for separating temporary working context, durable user knowledge, and source-preserving documents.

Agent teams often begin with one innocent word: memory. A recent message is called memory. A saved preference is called memory. A retrieved paragraph from a policy manual is also called memory. All three may belong in the next model call, but they have different owners, lifetimes, failure modes, and deletion rules.

The architecture becomes easier to reason about when context, durable memory, and source documents are separate systems that meet only at retrieval time. The point is not taxonomy for its own sake. The point is knowing which record is authoritative when an answer is wrong.

The short version

Context is the temporary working set for one request: recent turns, tool results, an active plan, and intermediate artifacts.
Memory is a durable conclusion that should influence future decisions: a preference, constraint, relationship, result, or corrected fact.
Documents preserve evidence: policies, manuals, contracts, research, transcripts, and other sources that must remain inspectable.
Retrieval composes these layers into a bounded prompt. It does not turn them into one store.

Start with lifecycle, not storage technology

A vector database does not tell you what deserves to persist. A graph does not tell you who may correct a fact. A larger context window does not tell you which source must survive deletion or re-indexing. Those are lifecycle questions, and they should be answered before choosing an index.

Layer	Typical contents	Authority	Normal lifetime	Primary failure
Working context	Recent messages, tool output, plan state	Current request assembly	One turn or workflow	Token bloat or stale task state
Durable memory	Preferences, constraints, decisions, corrections	Canonical scoped record and history	Until corrected, expired, or deleted	Wrong ownership or stale recall
Source corpus	Manuals, policies, contracts, long-form evidence	Original bytes or exact source text	Retention policy of the source	Lost provenance or irrecoverable extraction

Context is a compiled working set

Context exists to help the model complete the current request. It may include the last few turns, a tool response, a selected document excerpt, and a small set of memories. The application should be able to explain why each item was included and to discard it when the task ends.

Three practices keep context useful:

Budget by role. Reserve space separately for instructions, current conversation, retrieved evidence, durable constraints, and the model response.
Prefer summaries only when their source remains available. A compact task summary is useful; it should not silently replace the trace needed to debug it.
Rebuild context every turn. Do not append forever. Re-evaluate relevance, validity, and scope against the new request.

A long context window delays these decisions but does not remove them. Old tool results can contradict new results, early constraints can be buried, and cost grows even when most of the retained text is irrelevant.

Memory is a deliberate durable conclusion

A memory should be compact enough to change a later decision and structured enough to correct. At minimum, it needs content, structural scope, provenance, a stable identity, timestamps, and an immutable history of meaningful changes.

Scope is not decoration. A user preference belongs to that user. An agent operating rule may belong to an agent. A temporary workflow result may belong to a run. The authenticated project or namespace remains the outer security boundary; it should never be accepted from an untrusted request body.

A useful test: if the application cannot say who owns a record, how it was produced, and how to supersede it, it does not yet have a durable memory contract.

When the application already knows the exact record, it should write that record without another inference pass. When a conversation needs extraction, inference should produce a visible operation with an observable success or failure state. Quietly storing raw chat after extraction fails creates a second, undocumented product behavior.

Documents preserve evidence, not profiles

A handbook paragraph is not a user preference. A signed contract is not an inferred fact. Long sources need their own corpus because their authority comes from preservation. Store the original asset or exact textual source, record its checksum and identity, and treat Markdown, chunks, embeddings, entity graphs, and summaries as replaceable projections.

This separation makes several operations possible:

re-run extraction after a parser improves;
change chunk sizes without losing the source;
return a match with its document and neighboring evidence;
delete the full source family rather than leaving orphan vectors;
prove which version supported an answer.

Turning every useful document sentence into a profile memory is tempting, but it blurs retention, access control, and correction. Keep sources intact; derive only the smaller records the product genuinely needs across tasks.

Retrieval is where the layers meet

The prompt for one turn is a compiled artifact. A practical assembly sequence is:

bind the authenticated namespace and the request's user, agent, and run scopes;
identify the task and the kinds of evidence it can use;
retrieve active durable memories with scope and validity filters;
retrieve source excerpts when the request needs external or long-form evidence;
deduplicate overlapping material and preserve source identifiers;
order hard constraints, current task state, evidence, and optional background deliberately;
log enough retrieval evidence to reproduce a bad answer without logging secrets.

The result may look like one block of text to the model, but the application should retain the boundaries. A correction to a memory must not rewrite a contract. Deleting a document must not erase an unrelated user preference. Expiring a run must not delete a durable operating rule.

Four failure modes expose a collapsed architecture

Everything is appended to chat history

Costs rise, earlier constraints become hard to find, and a single malformed tool result can pollute every later turn. The system cannot distinguish expiry from deletion because nothing has its own lifecycle.

Every extracted sentence becomes memory

The store fills with transient observations, duplicates, and unsupported conclusions. Search may look relevant while returning facts the user never intended to preserve.

Chunks become the only copy of a document

Future parser and chunking improvements start from already-lost information. Citation becomes a label on a fragment rather than a path to an inspectable source.

Derived profiles become authoritative

A summary or profile is useful for progressive disclosure, but it can lag or fail to rebuild. It should cite canonical records and keep the previous valid view when refresh fails rather than replacing source truth.

A rollout sequence that stays debuggable

Instrument the current prompt. Measure tokens by source and identify material that is repeated without helping decisions.
Define structural scopes. Write down what user, agent, run, project, and document ownership mean in the product.
Introduce explicit durable writes. Begin with deterministic preferences and constraints before adding inferred extraction.
Preserve source documents. Make chunk and index rebuilds routine before depending on them for production answers.
Evaluate the complete answer loop. Inspect stored state, retrieved evidence, the final answer, latency, and failure behavior.

Where FishMem draws the boundary

FishMem keeps canonical memory records and source-preserving documents as separate product surfaces. Canonical records carry scope, history, temporal fields, and provenance. Document versions retain exact source content while retrieval chunks and indexes remain rebuildable. The application decides how both are assembled into the model's working context.

This is an architectural choice, not a claim that every application needs every layer. A short-lived tool may need only context. A personal assistant may need context and scoped memory. A support or research agent usually needs all three.