CONCEPT When the primary path fails, don't fail the request
Every model call can fail transiently — a 529 overloaded_error (the API is temporarily
overloaded across all users), a 429 rate_limit_error, or a 500 api_error. Those are the
retryable status codes; the Anthropic SDKs already retry them with exponential backoff (twice by
default, honoring the retry-after header). A fallback is the next layer: after retries
are exhausted, route the same request down an alternate path instead of surfacing an error to the user.
CONCEPT Four fallback tiers, cheapest degradation last
Order your fallbacks from "closest to the original answer" to "most degraded but still useful." Each rung trades a little quality or freshness for a much higher chance of returning something.
| Fallback tier | What changes | When to use | Cost of the swap |
|---|---|---|---|
| Another model tier | Opus → Sonnet → Haiku | Repeated 529 / overload on the primary model | Lower capability, cheaper & often less loaded |
| Another region / provider | 1st-party API → Bedrock / Vertex | Whole-provider outage or regional saturation | Feature-parity gaps; separate quota pool |
| Cached / canned response | Serve a prior or templated answer | Everything upstream is down | Stale, but instant and free |
| Degraded mode | Skip the LLM; return a rule-based / partial result | Last resort — keep the product usable | No generation, but no error page |
fallbacks parameter (beta) that
re-runs a refused request on a fallback model inside the same call — one round trip, no client logic. For
overload/region failover you own the routing client-side, but the mental model is identical: name the plan B up
front.DIAGRAM The fallback ladder
(retryable)"| R{"Retries left?
(backoff)"} R -->|"yes"| P R -->|"exhausted"| T2["🔵 Another model tier
Sonnet / Haiku"] T2 -->|"still failing"| T3["🟠 Another region / provider
Bedrock / Vertex"] T3 -->|"still failing"| T4["🟠 Cached / canned response"] T4 -->|"nothing cached"| T5["🔴 Degraded mode
rule-based / partial"] classDef ok fill:#d4edda,stroke:#28a745,color:#155724; classDef alt fill:#cce5ff,stroke:#4a90d9,color:#0a3a6a; classDef gate fill:#ffe8cc,stroke:#ff9f40,color:#9a4a0a; classDef bad fill:#f8d7da,stroke:#dc3545,color:#721c24; class DONE ok; class T2 alt; class T3 gate; class T4 gate; class T5 bad;
CODE A fallback ladder around the Messages API
Retryable errors (429/500/529) climb the ladder; a bad request (400) is not retryable and re-raises.
# Cheapest-degradation-last ladder. SDK already retries each call twice; # this handles what to do AFTER a path is exhausted. TIERS = [ ("claude-opus-4-8", primary_client), # 1st-party, top tier ("claude-sonnet-5", primary_client), # cheaper, often less loaded ("claude-opus-4-8", bedrock_client), # different region / provider ] def answer_with_fallback(messages): for model, client in TIERS: try: return client.messages.create( model=model, max_tokens=1024, messages=messages) except (anthropic.RateLimitError, # 429 anthropic.InternalServerError, # 500 anthropic.APIStatusError) as e: # 529 overloaded if getattr(e, "status_code", None) == 400: raise # not retryable — real bug log(f"tier {model} failed: {e}; trying next") return cached_or_degraded(messages) # canned answer, marked degraded
TRADE-OFFS Falling back vs. failing fast
✅ Layered fallbacks
- Availability survives a model/provider outage
- User sees an answer, not an error page
- Cheaper tiers absorb overload spikes
⚠️ Fallback everywhere, always
- Silent quality drops hide real incidents
- Falling back on a 400 masks a genuine bug
- Extra latency & cost per attempt; parity gaps across providers
EXAM TRAPS
overloaded_error is temporary capacity
pressure. Back off and retry, then drop to a cheaper tier or another region — don't error the whole request.retry-after before escalating.PRACTICE
Q1 · Your Opus calls start returning 529 during a traffic spike. What's the fallback order?
Retry with backoff first (the SDK does this, honoring retry-after). If it persists,
drop to a cheaper/less-loaded model tier (Sonnet/Haiku), then another region or provider
(Bedrock/Vertex), then a cached/canned response, and finally a degraded rule-based path — never a raw
error.
Q2 · Why should a 400 error not trigger a fallback?
400 invalid_request_error is non-retryable — the request itself is malformed (bad
schema, unsupported param). Falling back re-sends the same broken request down another path and hides the real bug.
Retryable codes are 429, 500, 529 (plus connection/timeout errors).
Q3 · What's the difference between the SDK's built-in retry and a fallback?
Retry = same path, wait, try again (SDK does this ~2× with backoff for 429/5xx). Fallback = a different path (cheaper model, other region, cache, degraded mode) once retries are exhausted. Retry handles blips; fallback handles a sustained outage of the primary path.