D1 · Agentic Architecture › Cluster 6 · Reliability

6.6 · Idempotency & Side-Effect Safety

Retries can double-execute — make tools safe to re-run

CONCEPT Retries and side effects collide

The whole reliability story so far — SDK auto-retry (6.3), fallback (6.4), even a client re-issuing a request after a timeout — assumes re-running a call is safe. For a read, it is. For a write with side effects (charge a card, send an email, create an order), a retry can double-execute: the first call succeeded but the response was lost, so the retry charges the customer twice.

⚠️ The dangerous case isn't "the call failed." It's "the call succeeded but the acknowledgement was lost." The caller can't tell the difference, retries, and the side effect happens again.
PropertyMeaningSafe to retry?
IdempotentRe-running yields the same end state (GET, PUT to a fixed key, deterministic upsert)🟢 Yes — retry freely
Non-idempotentEach run adds an effect (POST that creates, charge, increment, send)🔴 No — needs an idempotency key first

DIAGRAM How a lost ack causes a double charge

sequenceDiagram participant A as Agent participant P as Payments API A->>P: POST charge $50 (attempt 1) P->>P: 💳 charge succeeds P--xA: ✗ response lost (timeout) Note over A: looks like a failure → retry A->>P: POST charge $50 (attempt 2) P->>P: 💳 charges AGAIN — double execute! Note over A,P: Fix: send the SAME idempotency key
→ server dedupes attempt 2

CODE The idempotency-key pattern

Generate the key once per logical operation and send it on every retry of that operation. The server records the key and returns the original result instead of re-executing.

import uuid

# One key PER LOGICAL OPERATION — reused across ALL retries of it.
def charge_card(customer, amount):
    key = str(uuid.uuid4())                 # generated ONCE, outside any retry loop
    return _post_with_retry(key, customer, amount)

def _post_with_retry(key, customer, amount, attempts=3):
    for _ in range(attempts):
        try:
            return payments.post(
                "/charges",
                headers={"Idempotency-Key": key},   # SAME key each attempt
                json={"customer": customer, "amount": amount},
            )
        except TransientError:
            continue                            # server dedupes on the key → no double charge
    raise RuntimeError("charge failed after retries")
🔑 The magic is generate the key once, reuse it on every retry. A fresh UUID per attempt defeats the whole mechanism — the server sees each attempt as a new charge.

Design tools to be inherently safe

Where you own the tool, prefer designs that are naturally re-runnable:

EXAM TRAPS

Trap: "The SDK retries automatically, so all my tools are retry-safe." → The SDK retries the Anthropic API call. Your tool's side effects (a POST that charges) are your responsibility to make idempotent.
Trap: Blindly retrying a non-idempotent POST. → That's exactly how you double-charge. Add an idempotency key or confirm the operation is idempotent before retrying.
Trap: Generating a new idempotency key on each retry. → Defeats deduplication — the server treats each attempt as distinct. Generate once per operation, reuse across retries.
Trap: Assuming a failed response means the effect didn't happen. → A lost ack after a successful write is the classic double-execute trap. Idempotency keys make this safe.

PRACTICE

Q1 · An agent's "send_invoice_email" tool times out. Retry as-is?

Not safely — sending email is non-idempotent (each retry sends another). Add an idempotency key (or a per-invoice "already sent" guard) so a retry after a lost ack doesn't send twice.

Q2 · Why is generating the idempotency key inside the retry loop a bug?

Each attempt would carry a different key, so the server can't recognise attempt 2 as a retry of attempt 1 — it executes both. The key must be generated once per logical operation and reused on every retry.

Q3 · Which of these are safe to retry without special handling: GET user, PUT profile to a fixed key, POST create-order?

GET (read, no effect) and PUT to a fixed key (idempotent — same end state) are safe. POST create-order is not — it creates a new record each time; make it idempotent (upsert by cart key or an idempotency key) before retrying.