
How to evaluate agent memory before it reaches production
July 29, 2026Research

A deployment decision based on users, data authority, operating burden, collaboration, and recovery—not a generic feature checklist.
The deployment question is often framed as control versus convenience. That is too vague to produce a durable decision. The useful question is: who owns identity, data recovery, model operations, upgrades, incidents, and cost when the memory loop fails?
FishMem Desktop, the open-source service, and FishMem Cloud share core memory semantics, but they are different operating products. Choosing between them is choosing an authority boundary.
| Responsibility | Desktop | Self-hosted | Cloud |
|---|---|---|---|
| Primary operator | Individual user | Your platform team | FishMem service plus your application team |
| Data authority | Local SQLite and user backups | Your configured stores and object storage | Hosted workspace resources |
| Model operations | Local multilingual embeddings; no chat LLM | Your providers, limits, and credentials | Managed inference and extraction paths |
| Identity | One local agent boundary | Your API keys, users, and projects | Hosted accounts, projects, organizations, and keys |
| Recovery | User exports and restores snapshots | Your backups, restore drills, and repair jobs | Managed operations plus portable namespace export |
| Billing | None in the product | Your infrastructure and provider bills | FishMem subscription and usage ledger |
Desktop is designed for Codex and Claude Code workflows on one machine. The app owns one private local service and one SQLite/libSQL authority. The agent skill distills durable content; Desktop writes it with inference disabled. Embeddings run through a local quantized multilingual E5 model.
That narrow boundary is a feature. There is no hosted fallback, remote LLM extraction, organization layer, webhook service, or usage billing. The user owns the device, app updates, disk health, and backup location.
Desktop is a poor fit when several application servers need concurrent access, when teammates require shared policy and audit, or when recovery cannot depend on one person's device.
The Apache-2.0 engine can be embedded directly, and the open-source service provides a larger API and dashboard surface. You choose graph and vector adapters, provider credentials, persistent storage, network boundaries, and deployment topology.
The same freedom creates a concrete responsibility list:
Self-hosting makes sense when these controls are product requirements or existing platform capabilities—not simply because open source appears cheaper on a pricing page.
FishMem Cloud adds hosted projects and organizations, API keys, asynchronous inference workers, document extraction, managed object and vector resources, usage evidence, subscription billing, and hosted recovery operations around the same core contracts.
Your application still owns product policy: what to remember, user consent, how context is assembled, and which answers require additional verification. Managed infrastructure does not decide those things for you.
Before choosing, write down the failures your product must survive.
| Failure | Question to answer |
|---|---|
| Device lost | Where is the independent backup and who has tested restore? |
| Provider unavailable | Does work queue durably, fail closed, or degrade? |
| Index corrupt or stale | Can projections rebuild from canonical records and sources? |
| Credential revoked | Who sees the error and how is access restored? |
| Bad release | Can schema and application roll back without losing acknowledged writes? |
| Tenant isolation defect | Which boundary blocks cross-project data before retrieval? |
| Operator unavailable | Is the system still supportable and recoverable? |
Compare more than subscription price. Include engineering time for setup, upgrades, on-call, storage, model calls, backups, compliance evidence, and incident recovery. Also include the cost of delayed product work when the same team owns both the agent and its memory infrastructure.
For Desktop, include the user's time and the risk of untested backups. For Cloud, include usage and vendor dependency. For self-hosting, include the operational work that a managed plan would absorb.
A local app does not become a multi-user service because it exposes a CLI. Its trust and availability boundary remains the signed-in device.
You still depend on databases, model providers, libraries, and the engineers who operate them. Control changes the dependency graph; it does not remove it.
The service can operate the memory layer, but your application still owns consent, retention, risky actions, and how retrieved context affects users.