Agents Need Idempotency Keys

A billing agent runs on Claude Sonnet 5.5. Its Stripe tool has a 30-second HTTP timeout and two SDK retries. Step 4 of the plan reads “send invoice 12 to the customer”, and the runtime turns that into POST /v1/invoices/in_0012/send, which emails the invoice.

Stripe sends the email. The response is lost on the way back, the network stays down through both SDK retries, and the client gives up. The runtime hands the model a timeout. The model re-plans from a state that records no attempt, and writes a new step: “email invoice 12 to their billing address”. The customer gets the invoice twice.

The model did nothing wrong. The agent had no way to know whether the first send happened. The fix is an idempotency key, and it belongs in the runtime, where it holds on every run.

Why a timeout tells you nothing about the send #

An operation is idempotent if doing it twice has the same effect as doing it once. Reading a record is idempotent. Sending an email is not. A timeout only means no answer arrived in time, and two different failures look identical from the caller’s side:

So a retry of anything that isn’t idempotent needs help from the receiver.

Agents retry in more places than services do #

A normal service retries in one place: the HTTP client, a few times, after a timeout. An agent has at least five.

1. The HTTP client or SDK retries after a timeout.
2. The tool returns an error, and the model calls it again.
3. The model re-plans, and the new plan contains a step that already ran.
4. The run is replayed from a saved trace after a crash.
5. A person sees no result and clicks “try again”, starting a new run.

Paths 2 and 3 add a twist. The model rewrites the action in different words, so anything comparing request text sees two actions. Telling the model to check first mostly works, and across thousands of runs a small miss rate still sends a steady trickle of duplicate invoices.

What the key buys, and what it does not #

An idempotency key is a string the caller attaches to a request. The receiver stores the result of the first request carrying that key, and returns it for later ones.

curl -X POST https://api.stripe.com/v1/invoices/in_0012/send \
  -u "$STRIPE_SECRET_KEY:" \
  -H "Idempotency-Key: $KEY"

Stripe’s rules for its v1 API, which covers this endpoint, are worth reading closely:

• It saves the status code and body of the first request for a key, whether it succeeded or failed, once the endpoint has begun executing. Later requests with that key get the same result, 500 errors included. A replay carries the header Idempotent-Replayed: true.
• It compares the parameters with the original request and returns an idempotency_error if they differ.
• Keys are up to 255 characters, and should not contain email addresses or other personal identifiers.
• Keys are kept for at least 24 hours and may be pruned after that. A key reused after it is pruned starts a new request.

That first rule cuts both ways. A stored 500 is the one case where the key gives you nothing. Stripe says to treat a 500 as indeterminate, and advises against retrying with a fresh key until that outcome is resolved, because the original may have produced side effects. A stored 500 is therefore an unknown, and it has to be settled by looking at the world.

A stored 4xx is different. The request ran and failed, and the correction needs a new key. Some 4xx errors are never stored: a failed validation, a 429, or a 409 from a request still running under the same key. Retry a 409 or 429 later with the same key and parameters, because a new key there can start a second send. When a correction does need a new key, do not let it be random. Put an explicit attempt counter in the intent, record it in the ledger, and increment it when a human or a rule decides the request should change.

Derive the key from the intent, not the attempt #

Stripe suggests V4 UUIDs, and its SDKs can generate one and reuse it across their own retries of a call. That covers path 1 only. The re-plan in path 3 is a new call with a new key, and Stripe sees a new action. For retries across calls, Stripe says to persist one key per operation, and it lists deriving the key from an object, such as a cart ID, as a second strategy.

So the runtime, not the model, resolves each step into a typed call and hashes it:

import hashlib, json

def intent_key(tool: str, **fields: str) -> str:
    """Same intent, same key, however the model worded the step."""
    canonical = json.dumps({"tool": tool, **fields}, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(canonical.encode()).hexdigest()

# "send the invoice to the customer", resolved by the runtime to IDs
first = intent_key("send_invoice", invoice="in_0012", customer="cus_A17")
# "email invoice 12 to their billing address", resolved the same way
replan = intent_key("send_invoice", customer="cus_A17", invoice="in_0012")
# the same call with the run ID mixed in, after the user clicks "try again"
new_run = intent_key("send_invoice", invoice="in_0012", customer="cus_A17", run="run_2")

print(first[:16], first == replan, first == new_run)

ad0564ee9b9676dd True False

sort_keys=True makes field order irrelevant, so both wordings produce ad0564ee…. Leave the run ID out: path 5 starts a new run, so a key containing it changes on exactly the retry a person triggers.

Your ledger never expires, though, and Stripe’s keys do. A perfectly legitimate send of the same invoice six weeks later hashes to ad0564ee… too, and your own table blocks it forever. The distinguishing field has to come from somewhere, and it must not be the model. Take it from whatever caused the action: the inbound customer message ID, or the occurrence ID of the scheduled job. Every re-plan and every “try again” inside one cause lands on one key, a genuinely new request gets a new one, and the model has no field it can invent to slip past dedup. With cause="occ_0412" added, the key for step 4 becomes 66382ae1…, used from here on.

When the model writes part of the payload #

Sending an invoice has no body, which is why the example above is easy. Most agent actions are not. An agent that emails through Resend’s POST /emails writes the subject and body itself, and a re-plan writes them differently.

Hash only the identity fields: who, about what, caused by which event. Never the prose. Then store the resolved request alongside the key, and on every retry re-send the stored request, not the one the model just wrote. Otherwise the same key arrives with a different body, and Resend answers 409 invalid_idempotent_request while Stripe answers with an idempotency_error.

Treat that 409 as information, not an error. It means this action was attempted, and the right response is to settle it from your own record. Resend’s other 409, concurrent_idempotent_requests, means the first request is still running: retry later with the same key. What the model must never see is a bare error, because a bare error has one obvious next move.

The ledger, written before the call #

Stripe may drop a key once it is 24 hours old, and your runtime can crash between sending a call and reading the answer. So keep your own record, keyed by the same hash and written first:

create table agent_actions (
  key         text primary key,
  tool        text not null,
  target      text not null,
  attempt     int  not null default 1,
  state       text not null check (state in ('pending','done','failed','unknown')),
  request     jsonb not null,         -- the resolved call, re-sent on every retry
  result      jsonb,                  -- what came back, returned to later callers
  owner       text,                   -- the worker holding the lease
  lease_until timestamptz,            -- no other worker settles the row before this
  created_at  timestamptz not null default now(),
  updated_at  timestamptz not null default now()  -- the default covers inserts; set it on every update
);

insert into agent_actions (key, tool, target, state, request, owner, lease_until)
values ('66382ae1…', 'send_invoice', 'in_0012', 'pending',
        '{"path":"/v1/invoices/in_0012/send"}', 'worker-3', now() + interval '5 minutes')
on conflict (key) do nothing
returning key;

on conflict do nothing with returning says whether this call inserted the row. Commit that insert before the call, or a crash rolls it back. When no row comes back, a plain select reads the state, and the state picks the branch:

• A row came back. First attempt. Send request with the key, then write result and set the state to done, failed or unknown. A 409 or 429 leaves it pending.
• No row came back, state done. Return the stored result. Nobody calls anything.
• No row came back, state pending, unknown, or failed with a stored 500. Once the lease has expired, settle it: re-send the stored request with the stored key within 24 hours of the first send and read the response, or look for the effect in the world. Then write the state.
• No row came back, state failed with any other 4xx. The request failed. A corrected one needs a new key, so bump attempt and hash again.

Written after the call, the ledger is only a log: if the process dies in between, the world has an effect your runtime doesn’t know about. Written before, a restarted runtime finds every unsettled row and knows exactly which calls to chase.

A pending row can also mean the first call is still running, in another worker or behind a slow network. Settle a row only after lease_until passes, and make the lease outlast every attempt: with a 30-second timeout and two retries, well over 90 seconds. A Stripe re-send that still races the original gets a 409, which saves nothing and only means try again later. A keyless tool has no such guard. An expired lease does not prove its call stopped, so its row goes to unknown instead of a re-send.

Give the model a third outcome #

Not every API takes a key. Stripe does on every v1 POST, Resend on POST /emails, POST /emails/batch and three Inbox endpoints, and many tools do not. Stripe has no lookup by key, but each event caused by an API request records that request’s idempotency key, and events can be listed for 30 days, so a send that happened shows up as an invoice.sent event carrying the key. Re-sending the request and reading Idempotent-Replayed works too, inside the key’s window, but it is not read-only: if the original never ran, the re-send performs it. A tool with neither a key nor a readback leaves one accurate state after a timeout. Say so in the tool result:

{"status": "unknown", "key": "66382ae1…", "action": "send_invoice in_0012"}

A model handed a plain error tries again. An unknown carrying the key tells the model, and the person reviewing the run, exactly which action needs checking. Replays work the same way. When a run resumes from a saved trace, re-send the stored request with its stored key rather than asking the model to plan the step again. That needs each step stored as typed data.

Back to invoice 12 #

Same agent, same 30-second timeout, same lost response. The runtime committed 66382ae1… before the call, so the re-planned step, from the same cause, resolves to that same row, inserts nothing, and never reaches Stripe. If the row is still pending once its lease runs out, the runtime re-sends the stored request with the stored key and Stripe replays the original send. The model’s second attempt was never the problem. It was the only party that had no way to tell the two timeouts apart, and now it doesn’t have to.