What is agent memory? A production architecture for context, memory, and sources

An agent can finish a difficult task and still begin the next run as if nothing happened. The model may be capable, the tools may work, and the context window may be large, but none of those things decide what should survive after the current request ends.

Agent memory is the system that makes that decision explicit. It preserves durable conclusions, retrieves the relevant ones for a later task, and keeps enough history and provenance to correct them when reality changes.

The short version

• Working context is the temporary input assembled for one model call or workflow.
• Agent memory is durable, scoped knowledge that should influence later decisions.
• Source documents preserve inspectable evidence such as manuals, policies, transcripts, and contracts.
• Retrieval selects a small, relevant subset from memory and sources for the current task.
• A production memory system also needs ownership, correction, deletion, history, retries, and observable failure.

Agent memory is durable decision state

A useful memory is not merely text that existed in a previous conversation. It is a durable record that can change a future action. Examples include a user preference, a project constraint, a verified support resolution, a corrected account fact, or a decision that a coding agent should respect in the next session.

That definition creates a practical test:

If a record cannot explain who owns it, why it was stored, when it is valid, and how it can be corrected or deleted, it is not yet a production memory contract.

Some memories are extracted from conversations. Others are already known and should be written verbatim. FishMem makes that distinction explicit: the default inferred path creates a durable asynchronous task, while infer: false stores already-distilled content without another model call.

Why chat history is not long-term memory

Chat history is useful because recent turns carry local intent, references, and conversational rhythm. It becomes a poor long-term store when an application keeps appending forever.

Three failures appear quickly:

• Cost grows with age. The application repeatedly pays to send old turns even when most of them do not affect the current task.
• Important facts become hard to find. A durable constraint may be buried beneath tool output, acknowledgements, and obsolete plans.
• Correction becomes ambiguous. The transcript contains both the old claim and the correction, but it does not define which one is current.

A short conversation window and durable memory solve different problems. Keep enough recent turns for the immediate exchange, then retrieve only the older conclusions that matter now.

Why a larger context window does not replace memory

A larger window increases how much an application can send to a model. It does not decide what deserves to persist, who may access it, or which version is authoritative.

Even when the full history fits, the application still needs to answer:

• Which user, agent, project, or run owns this fact?
• Should the fact expire, remain active, or be superseded?
• Was it asserted by a user, inferred by a model, or copied from a source?
• How can a user inspect, update, or delete it?
• Which records should be included in this request rather than every request?

Context is the compiled working set. Memory is one of the durable inputs used to build it.

Why RAG and memory are different

Retrieval-augmented generation is usually built around a corpus of source material. The authority comes from the source: a handbook, product manual, policy, research paper, or customer document. The system retrieves passages because the answer should remain grounded in inspectable evidence.

Agent memory is usually smaller and more dynamic. It tracks conclusions such as preferences, constraints, relationships, outcomes, and corrections. These records may change frequently and often need explicit history.

A production agent commonly needs both:

• Use source RAG when the answer must cite or inspect a document.
• Use durable memory when a conclusion should influence later behavior.
• Keep the source available when a memory was derived from evidence.
• Do not turn every retrieved paragraph into a user profile fact.

FishMem keeps canonical memory records separate from source-preserving documents. The application can retrieve both into one prompt without pretending they have the same lifecycle.

The four layers of a production memory architecture

1. Working context

This is the request-local state: recent messages, tool results, an active plan, retrieved memories, and source excerpts. It should be rebuilt for each turn instead of appended without a limit.

2. Canonical memory records

These are durable conclusions with stable identity, scope, timestamps, metadata, and change history. A canonical record should remain understandable without replaying the full conversation that created it.

3. Source corpus

This layer retains original or lossless source content. Chunks, embeddings, summaries, and graphs are projections that can be rebuilt when extraction or retrieval improves.

4. Retrieval and context assembly

Retrieval selects candidates; context assembly decides how to use them. A useful assembly step deduplicates overlapping records, preserves source identifiers, separates hard constraints from optional background, and applies a token budget.

Scope is part of the data model

Agent memory becomes dangerous when scope is treated as a free-form tag. A caller should not be able to ask the model which tenant or user a request belongs to.

FishMem uses structural scopes:

• user_id for durable knowledge about an authenticated end user;
• agent_id for facts or operating knowledge owned by a particular agent;
• run_id for workflow-specific state that should not automatically cross runs;
• the authenticated project as the outer workspace isolation boundary.

The server should derive the project and user identity from trusted authentication state. The model may use retrieved memory, but it should not be allowed to choose the security boundary.

The write loop: observe, decide, persist

A reliable write loop separates accepting work from finishing it.

1. Observe an event. A conversation ends, a user corrects a fact, or a workflow reaches a verified outcome.
2. Choose the write policy. Use inference when the input needs extraction. Use a verbatim write when the application already knows the exact durable conclusion.
3. Bind scope and idempotency. The same retry should not create a second command or duplicate records.
4. Persist the canonical outcome. Store stable records and their history before treating derived search projections as complete.
5. Expose completion or failure. Asynchronous extraction should return a durable receipt that the application can observe.

Writing every message as memory is not a safe default. Acknowledgements, temporary plans, guesses, and raw tool output usually belong in working context or an audit trace, not a durable profile.

The recall loop: retrieve, filter, assemble

Recall begins with the current decision, not with a request to fetch everything known about a user.

1. Derive the trusted project and structural scopes.
2. Use the latest task or message as the retrieval query.
3. Apply validity, metadata, and ownership filters.
4. Retrieve a bounded number of relevant memories.
5. Retrieve source evidence separately when the task needs it.
6. Deduplicate and label the resulting context.
7. Record enough request evidence to debug a bad answer without logging secrets.

FishMem recall can combine semantic, keyword, temporal, and graph signals. The application still owns the final prompt contract and should keep retrieved memory visibly separate from instructions and source quotations.

How memory changes over time

Durable does not mean permanent or immutable. A user can move, a project can change databases, and a support workaround can become obsolete.

A mature system distinguishes:

• Update: the canonical content was wrong or needs an explicit correction.
• Supersession: an earlier fact was true, but a newer fact is now active.
• Expiration: a fact should stop participating after a known boundary.
• Deletion: the record must be removed from active recall under the product retention contract.

FishMem records history so an application can inspect how a memory changed. Its bi-temporal model can represent both when a fact was true and when the system learned it, which is different from simply overwriting a row.

What should an agent remember?

Store information that is likely to improve a later decision and can be owned and corrected clearly:

• stable user preferences and accessibility needs;
• verified project decisions and constraints;
• support outcomes and approaches that already failed;
• account relationships and commitments;
• corrections that must supersede an earlier conclusion.

Avoid automatically storing secrets, complete transcripts, temporary tool output, unsupported model guesses, or document passages whose authority belongs to the original source.

Where FishMem fits

FishMem provides the durable memory and source-document layers, plus the APIs needed to add, search, inspect, update, delete, and observe them. The open-source engine is available for teams that want to operate the system themselves. FishMem Cloud adds managed infrastructure, project API keys, workspace isolation, usage metering, webhooks, and a dashboard.

FishMem does not replace the application agent loop. Your application still decides when a workflow has produced a durable conclusion, how much recalled context to include, and what the model may do with it.

Frequently asked questions

Is agent memory just a vector database?

No. A vector index can find semantically similar text, but a production memory system also needs canonical records, structural scope, correction history, temporal validity, deletion, retries, and observable operations.

Should every conversation be written to memory?

No. Persist only the durable conclusions that should influence later work. Keep recent conversational detail in a bounded context window, and keep long-form evidence in a source corpus.

Does memory remove the need for RAG?

No. Memory and RAG have different authorities and lifecycles. Many support, research, and enterprise agents need both durable memory and source-backed document retrieval.

Can memory be local or self-hosted?

Yes. The correct deployment depends on who should own storage, recovery, access control, and operations. FishMem supports an open-source engine, a managed Cloud service, and a local Desktop surface with different operational boundaries.

Start with one memory loop

Choose one repeated decision, define the smallest durable record that would improve it, and test recall across two separate runs. The Cloud quickstart shows the first add and search calls, while the memory evaluation cookbook turns the loop into a release gate.

For a deeper treatment of the boundaries, read Memory is not context. You can also inspect the open-source FishMem engine or create a free FishMem Cloud workspace.