CONCEPT Grow the wait, cap the attempts
When a retryable error (429/5xx) comes back, the worst thing you can do is retry immediately in a tight loop — you hammer an already-strained service and make the overload worse. Exponential backoff waits a delay that doubles each attempt (1s, 2s, 4s, 8s…), giving the service room to recover. Three guardrails make it production-safe: cap the delay (e.g. never wait more than 60s), cap the attempts (e.g. 5 tries, then give up and surface the error), and — the piece everyone forgets — add jitter.
delay = min(cap, base · 2^attempt), then add jitter,
then sleep. Stop after a fixed number of attempts.CONCEPT Jitter defeats the thundering herd
Imagine 500 clients all get a 529 at the same instant. With pure exponential backoff they all wait exactly 1s, then all retry at the same moment — a synchronized stampede that re-overloads the service. Then they all wait 2s and stampede again. This is the thundering herd. Jitter — adding a random component to each delay — spreads the retries across a window so they arrive smoothly instead of in a spike. It's the single most important addition to backoff, and the reason "exponential backoff with jitter" is the phrase, not "exponential backoff" alone.
sleep(random(0, min(cap, base·2^n)))
— spreads retries maximally; equal jitter — half the delay fixed, half random. Both break the synchronized herd.CONCEPT The retry-after header overrides your math
A 429 response includes a retry-after header telling you exactly how many
seconds to wait before the request can succeed. When it's present, honor it — it's the server's
authoritative instruction, and retrying earlier is guaranteed to fail. Treat retry-after as the
floor for that attempt's delay; use your computed backoff only when the header is absent (typical for
500/529). Anthropic's SDKs already do this: they auto-retry 408/409/429/5xx with exponential backoff, twice by
default, honoring retry-after when present.
DIAGRAM One retry decision
surface the error"] N -->|"yes"| H{"retry-after header?"} H -->|"yes (429)"| USE["🟠 delay = retry-after"] H -->|"no (5xx)"| CALC["delay = min(cap, base·2^n)"] CALC --> J["🟢 + random jitter"] USE --> S["sleep(delay), retry"] J --> S classDef ok fill:#d4edda,stroke:#28a745,color:#155724; classDef gate fill:#ffe8cc,stroke:#ff9f40,color:#9a4a0a; classDef bad fill:#f8d7da,stroke:#dc3545,color:#721c24; class J ok; class S ok; class USE gate; class FAIL bad;
CODE Backoff with jitter and a retry-after floor
Capped attempts, capped delay, full jitter, and the header takes priority when present.
import time, random, anthropic def call_with_backoff(fn, max_attempts=5, base=1.0, cap=60.0): for attempt in range(max_attempts): try: return fn() except anthropic.RateLimitError as e: # 429 hdr = e.response.headers.get("retry-after") delay = float(hdr) if hdr else None # header wins except anthropic.APIStatusError as e: if e.status_code < 500: raise # terminal 4xx delay = None # 5xx: compute below if delay is None: delay = min(cap, base * 2 ** attempt) # exponential… delay = random.uniform(0, delay) # …+ full jitter time.sleep(delay) raise RuntimeError("exhausted retries") # capped — give up
TRADE-OFFS Fixed delay vs. exponential-with-jitter
✅ Exponential backoff + jitter
- Doubling delay gives the service room to recover
- Jitter spreads retries — no synchronized stampede
- Capped attempts fail cleanly instead of looping forever
⚠️ Fixed-interval retry (e.g. every 1s)
- Trivial to implement
- But: no jitter → all clients retry in lockstep
- Keeps hammering an overloaded service, deepening the outage
EXAM TRAPS
retry-after header and using your own delay on a 429. →
The header is the server's authoritative wait; retrying earlier just fails again. Honor it.PRACTICE
Q1 · Why add jitter to exponential backoff?
To prevent the thundering herd: without jitter, many clients that failed at the same instant back off by the same amount and all retry simultaneously, re-overloading the service. Random jitter spreads the retries across a window.
Q2 · A 429 returns retry-after: 12. Your backoff computed 3s. What do you wait?
12 seconds — the retry-after header is authoritative. Retrying before it
elapses is guaranteed to fail. Use your computed backoff only when the header is absent.
Q3 · Name the three guardrails on a backoff loop besides doubling the delay.
(1) Jitter — randomize each delay; (2) cap the delay so it doesn't grow unbounded; (3) cap the attempts so the loop terminates and surfaces the error instead of retrying forever.