Designing asynchronous memory writes that survive retries

August 5, 2026Guides12 min read

A production write protocol for receipts, idempotency, terminal state, retries, and accounting when model work outlives HTTP.

Memory extraction is not an ordinary database insert. A provider can accept work after the client times out. A queue can deliver the same message twice. A worker can commit records and crash before acknowledging delivery. If the API models all of that as one synchronous request, the client cannot know whether to retry, wait, or investigate.

A durable write protocol separates acceptance from completion. The HTTP request creates one command identity and returns a receipt. A persisted task owns execution. A terminal event owns the outcome.

The short version

Require an idempotency key before accepting inferred writes.
Persist the command and reserve hosted usage before model work begins.
Return HTTP 202 with a durable event identity, not an invented completed memory.
Treat queue delivery as a wakeup; the database task and lease are authoritative.
Commit canonical records exactly once and expose terminal success or failure.
Settle billing from the same durable outcome: commit once on success, refund terminal failure.

The ambiguity window

Consider a client that sends an inferred memory write and waits five seconds. The provider call takes six seconds. At second five the client sees a timeout. At second six the worker commits two memories. If the client retries without a stable identity, the system may repeat inference and create duplicates. If it refuses to retry, it may leave the user believing the write was lost.

The protocol must make these states distinguishable:

State	What it proves	Safe client action
Accepted	The command is durably recorded	Keep the receipt and observe the event
Pending	No worker currently owns a valid lease	Wait; repair or dispatch may reclaim it
Running	A worker owns an unexpired lease	Do not create a second command
Succeeded	Canonical records and required state committed	Read the result
Failed	Automatic execution reached a terminal error	Inspect error; authorize an explicit retry if appropriate

Commit the command before calling the model

FishMem Cloud validates the request, binds the authenticated workspace, hashes the normalized command, claims the idempotency key, reserves credits, and creates one memory_infer operation task. Only then is work dispatched. The HTTP response is 202 because the system has accepted responsibility for the command, not completed it.

The task payload contains the structural scope, inference input, idempotency identity, and hosted usage authorization needed to execute after the original request has disappeared. Successful completion replaces sensitive raw input in the durable task with a command fingerprint and the minimum result evidence needed for audit and replay.

Idempotency is a command contract

An idempotency key is not a duplicate-removal hint. It means one caller intent has one identity.

Same key and same normalized command: return or continue the original operation.
Same key and different command: return a conflict.
Different key and same text: treat as a separate intent unless the product explicitly defines otherwise.

Hash the semantic command fields, not transport noise. Scope, inference mode, message content, and memory options matter. A request ID, connection timestamp, or retry counter normally does not.

The queue is not the task store

Cloudflare Queue, a cron job, or any other dispatcher may wake a worker, but delivery cannot be the only evidence that work exists. Messages can be duplicated, delayed, or exhausted. The persisted task records attempts, status, lease owner, lease expiry, next attempt time, terminal error, and final result.

A worker claims a task with compare-and-swap semantics. If it crashes, another worker can reclaim the expired lease. If the queue message never arrives, scheduled repair can find a pending task and dispatch it again. Both paths converge on the same operation identity.

Freeze the prepared write plan

Retrying a model call can produce different facts. That may be acceptable before any canonical write exists, but it is unsafe after a partial commit. FishMem's core journal freezes stable record identities and the prepared mutation plan before completing projections. A retry replays the same plan rather than inventing a second interpretation.

The operation journal tracks canonical and projection status separately. If the record commit succeeds but a vector write fails, retry repairs the vector for the existing record. It does not add the memory again.

Terminal state must carry useful evidence

A client should not have to parse worker logs to learn the outcome. The Event API exposes a privacy-safe projection of the durable task with status, attempts, timestamps, error information, and result references. SDKs can provide a convenience wait helper, but the underlying resource remains observable by any client that keeps the event ID.

Errors should be stable enough to drive a decision:

invalid input: correct the request; do not retry automatically;
idempotency conflict: investigate caller identity reuse;
provider or temporary infrastructure failure: apply bounded retry;
authorization or billing failure: require a new valid authorization;
terminal extraction rejection: show that no memory was committed.

Billing belongs inside the protocol

Hosted inference cannot maintain separate truths for work and money. FishMem reserves the required credits before enqueueing. On terminal success, the reservation settles once. On terminal failure, it is released and the request ledger records the refund. A manual retry must re-authorize usage before executing; an old failed task is not an unlimited claim on future credits.

This avoids three bad outcomes: charging for work that never ran, refunding after a successful commit, or charging twice after an ambiguous retry.

Design the retry policy by failure class

Failure	Automatic retry?	Reason
Provider timeout before result	Bounded, if command is still safe	May be transient; idempotent task controls duplication
Invalid structured output	Only under an explicit extraction policy	Repeated prompts may not repair a semantic mismatch
Insufficient credits	No	Requires new authorization or plan change
Projection write failure	Yes	Repair the existing canonical record
Idempotency conflict	No	The caller reused one identity for two commands

What to test before release

Repeat the same request before, during, and after completion.
Reuse the key with a changed payload and confirm a conflict.
Kill the worker before inference, after inference, and after canonical commit.
Drop queue delivery and confirm repair finds the durable task.
Expire a lease and confirm only one new worker acquires it.
Force projection failure and verify retry does not duplicate canonical rows.
Verify success charges once and terminal failure refunds once.
Remove raw task input after success and confirm the event remains useful.

The API boundary

infer=false remains a synchronous deterministic write because the caller already supplies the durable record. infer=true is asynchronous by default because model work can outlive HTTP and fail independently. Conflating the two paths would either make deterministic writes unnecessarily complex or make inferred writes falsely synchronous.