D5 · Context & Reliability › Cluster 5 · Reliability: Errors & Retries

5.3 · Exponential Backoff + Jitter

Back off, spread out

CONCEPT Grow the wait, cap the attempts

When a retryable error (429/5xx) comes back, the worst thing you can do is retry immediately in a tight loop — you hammer an already-strained service and make the overload worse. Exponential backoff waits a delay that doubles each attempt (1s, 2s, 4s, 8s…), giving the service room to recover. Three guardrails make it production-safe: cap the delay (e.g. never wait more than 60s), cap the attempts (e.g. 5 tries, then give up and surface the error), and — the piece everyone forgets — add jitter.

🔑 The base formula: delay = min(cap, base · 2^attempt), then add jitter, then sleep. Stop after a fixed number of attempts.

CONCEPT Jitter defeats the thundering herd

Imagine 500 clients all get a 529 at the same instant. With pure exponential backoff they all wait exactly 1s, then all retry at the same moment — a synchronized stampede that re-overloads the service. Then they all wait 2s and stampede again. This is the thundering herd. Jitter — adding a random component to each delay — spreads the retries across a window so they arrive smoothly instead of in a spike. It's the single most important addition to backoff, and the reason "exponential backoff with jitter" is the phrase, not "exponential backoff" alone.

Two common jitter styles: full jitter — sleep(random(0, min(cap, base·2^n))) — spreads retries maximally; equal jitter — half the delay fixed, half random. Both break the synchronized herd.

CONCEPT The retry-after header overrides your math

A 429 response includes a retry-after header telling you exactly how many seconds to wait before the request can succeed. When it's present, honor it — it's the server's authoritative instruction, and retrying earlier is guaranteed to fail. Treat retry-after as the floor for that attempt's delay; use your computed backoff only when the header is absent (typical for 500/529). Anthropic's SDKs already do this: they auto-retry 408/409/429/5xx with exponential backoff, twice by default, honoring retry-after when present.

DIAGRAM One retry decision

flowchart TD A["Retryable error (429/5xx)"] --> N{"Attempts left?"} N -->|"no — cap hit"| FAIL["🔴 Give up,
surface the error"] N -->|"yes"| H{"retry-after header?"} H -->|"yes (429)"| USE["🟠 delay = retry-after"] H -->|"no (5xx)"| CALC["delay = min(cap, base·2^n)"] CALC --> J["🟢 + random jitter"] USE --> S["sleep(delay), retry"] J --> S classDef ok fill:#d4edda,stroke:#28a745,color:#155724; classDef gate fill:#ffe8cc,stroke:#ff9f40,color:#9a4a0a; classDef bad fill:#f8d7da,stroke:#dc3545,color:#721c24; class J ok; class S ok; class USE gate; class FAIL bad;

CODE Backoff with jitter and a retry-after floor

Capped attempts, capped delay, full jitter, and the header takes priority when present.

import time, random, anthropic

def call_with_backoff(fn, max_attempts=5, base=1.0, cap=60.0):
    for attempt in range(max_attempts):
        try:
            return fn()
        except anthropic.RateLimitError as e:              # 429
            hdr = e.response.headers.get("retry-after")
            delay = float(hdr) if hdr else None       # header wins
        except anthropic.APIStatusError as e:
            if e.status_code < 500: raise              # terminal 4xx
            delay = None                              # 5xx: compute below
        if delay is None:
            delay = min(cap, base * 2 ** attempt)     # exponential…
            delay = random.uniform(0, delay)          # …+ full jitter
        time.sleep(delay)
    raise RuntimeError("exhausted retries")     # capped — give up

TRADE-OFFS Fixed delay vs. exponential-with-jitter

✅ Exponential backoff + jitter

  • Doubling delay gives the service room to recover
  • Jitter spreads retries — no synchronized stampede
  • Capped attempts fail cleanly instead of looping forever

⚠️ Fixed-interval retry (e.g. every 1s)

  • Trivial to implement
  • But: no jitter → all clients retry in lockstep
  • Keeps hammering an overloaded service, deepening the outage

EXAM TRAPS

Trap: "Exponential backoff is enough." → Without jitter, synchronized clients back off in lockstep and re-stampede. Jitter (randomized delay) is what actually breaks the thundering herd.
Trap: Ignoring the retry-after header and using your own delay on a 429. → The header is the server's authoritative wait; retrying earlier just fails again. Honor it.
Trap: No cap on attempts. → An unbounded retry loop can hammer forever. Cap the attempts (and the max delay), then surface the failure.
Trap: Retrying a 400/404 with backoff. → Backoff is only for transient errors (429/5xx). A terminal 4xx never becomes valid by waiting — fix the request.

PRACTICE

Q1 · Why add jitter to exponential backoff?

To prevent the thundering herd: without jitter, many clients that failed at the same instant back off by the same amount and all retry simultaneously, re-overloading the service. Random jitter spreads the retries across a window.

Q2 · A 429 returns retry-after: 12. Your backoff computed 3s. What do you wait?

12 seconds — the retry-after header is authoritative. Retrying before it elapses is guaranteed to fail. Use your computed backoff only when the header is absent.

Q3 · Name the three guardrails on a backoff loop besides doubling the delay.

(1) Jitter — randomize each delay; (2) cap the delay so it doesn't grow unbounded; (3) cap the attempts so the loop terminates and surfaces the error instead of retrying forever.