How to evaluate agent memory before it reaches production

A memory benchmark can tell you that one frozen system answered one dataset under one harness. It cannot tell you whether your support agent will remember the right account, respect a corrected preference, avoid another tenant's data, and recover safely after a provider timeout.

Production evaluation therefore needs two layers: public benchmarks for broad failure categories and a private, versioned acceptance set built from the product's own traces.

The short version

• Evaluate the complete loop: write policy, persisted state, retrieval, context assembly, answer, latency, and failure behavior.
• Freeze code, models, prompts, storage adapters, data splits, and judge rules.
• Use development data for tuning and protect a held-out release set.
• Report item-level outputs, paired differences, uncertainty, and known failures—not one average score.
• Turn every production-shaped failure into a durable regression case.

Begin with the product decision

“Improve memory accuracy” is not an acceptance criterion. Name the decision that memory should improve. Examples include selecting the current shipping address, preserving a user's communication constraint, continuing a multi-session workflow, citing the policy version used for an answer, or abstaining when no supported memory exists.

Each case should contain the source trace, the expected durable records, the retrieval query, required and forbidden evidence, the expected downstream decision, and the reason that decision matters.

Measure the loop in layers

Layer	Core question	Example metric
Write	Did the system store the right durable information?	Required fact recall, forbidden fact rate, duplicate rate
State	Did corrections and validity behave correctly?	Current-value and as-of accuracy
Retrieval	Did the context contain the evidence needed to answer?	Context completeness, precision, rank
Answer	Did the agent make the correct decision from that evidence?	Task rubric or exact answer
Operations	Can the loop survive real delivery conditions?	Tail latency, failure rate, retries, index visibility
Safety	Was any forbidden scope or source exposed?	Cross-tenant leakage and policy violations

Context completeness comes before answer correctness

An answer can be correct for the wrong reason. The model may already know the answer, the evaluator may leak it in the question, or irrelevant retrieved text may coincidentally lead to the expected phrase. Inspect whether the assembled context actually contains the required evidence and excludes contradicted or unauthorized material.

Likewise, a relevant retrieval hit does not guarantee a correct answer. The application may order evidence poorly, omit temporal markers, or ask the model to perform a decision that belongs in deterministic code. Scoring both layers tells you where to fix the system.

Build a representative trace set

Sample the real distribution, then deliberately add hard cases. A useful acceptance set includes:

• single-session facts and preferences;
• multi-session continuations;
• facts that change over time;
• late-arriving corrections;
• similar users or projects that must stay isolated;
• requests where the correct behavior is to abstain;
• invalid writes, retries, deletion, and restore;
• long sources requiring citations;
• tool results that should expire instead of becoming durable memory.

Remove or redact sensitive user data while preserving the decision structure. Give each case a stable identity so results can be compared across releases.

Separate development and release decisions

Use a development split to inspect failures, adjust prompts, change retrieval, and tune thresholds. Keep a held-out set untouched until the candidate is frozen. Repeatedly checking the release set while tuning turns it into another development set.

For small changes, paired evaluation is more informative than comparing two independent averages. Score the same cases under both candidates, record which items flipped, and inspect whether gains cluster in one category while another regresses.

Freeze the moving parts

• source revision and dependency lock;
• memory mode and extraction prompt;
• answerer, embedder, reranker, and judge models;
• temperature and retry behavior;
• storage and vector adapters;
• chunking and top-k configuration;
• dataset split and source preprocessing;
• judge rubric and normalization code;
• raw item-level outputs and failure logs.

Without a manifest, a benchmark result is a screenshot. With a manifest and saved outputs, it is an experiment another engineer can challenge.

Read benchmark claims in their own harness

LoCoMo, LongMemEval, and BEAM expose different aspects of long-term memory. Their scores are not interchangeable, and numbers from different models, prompts, retrieval limits, or judge protocols should not be placed in one table as if they were a race.

FishMem's frozen 2026-08-24 portfolio compares every item with mem0 OSS 3.1.2 under the same disclosed configuration within each pair. On 500 LongMemEval oracle questions, FishMem scored 88.2% and mem0 83.8%, a +4.4-point paired delta with a 95% confidence interval from +1.0 to +7.8. On 400 BEAM 100K questions, FishMem scored 46.7% and mem0 41.0%, a +5.7-point delta with a 95% confidence interval from +1.8 to +9.5.

LoCoMo is the losing row, and it remains in the release result. Across categories 1–5 and 1,986 paired questions, FishMem scored 67.5% and mem0 71.1%, a −3.7-point delta with a 95% confidence interval from −5.8 to −1.5. LongMemEval oracle is not LongMemEval-S. Recovered process attempts make public cost and end-to-end wall-clock comparisons invalid for this run, while the winning FishMem runs also used substantially more retrieved context. Those limitations are part of the result, not footnotes to hide.

Add operational metrics

A memory system that is accurate only when every provider call succeeds is not ready. Track:

• write acceptance and completion latency;
• p50, p95, and p99 retrieval latency;
• context tokens per query and per successful answer;
• provider calls, retries, timeouts, and terminal failures;
• time from canonical commit to index visibility;
• duplicate and idempotency-conflict rates;
• rebuild and restore duration;
• scope-denial and leakage tests.

Use judges carefully

LLM judges are useful for semantic answers, but they introduce their own variance and bias. Keep the rubric narrow, save the judge explanation, blind system identity where practical, and manually audit a sample of agreements and disagreements. Exact checks remain preferable for IDs, scope, deletion, event state, and other deterministic contracts.

Do not retry a judged question until it passes and then report the best result. Define retry policy before the run and keep failures visible.

Turn failures into the roadmap

Group failures by the decision that broke: missing write, wrong scope, stale fact, poor ranking, incomplete context, answer reasoning, or operations. Fix the deepest recurring cause, add a regression case, and rerun the whole group. A higher score without an explanation of changed failures is difficult to trust and difficult to maintain.

Release checklist

1. Freeze the candidate and evaluation manifest.
2. Run the development set and inspect category-level regressions.
3. Run the held-out set once for the decision.
4. Verify persistence and projection state, not only final answers.
5. Run retry, restore, deletion, and cross-namespace journeys.
6. Publish or archive item-level outputs and limitations.
7. Define the production monitoring window and rollback threshold.

Further reading

• FishMem benchmark evidence and limitations
• FishMem benchmark runners
• Mem0: state of AI agent memory and open problems
• Zep: agent-memory testing and evaluation guides