D5 · Context & Reliability › Cluster 6 · Reliability: Degradation & Recovery

6.1 · Fallbacks (Model / Provider / Degraded)

A plan B for every call

CONCEPT When the primary path fails, don't fail the request

Every model call can fail transiently — a 529 overloaded_error (the API is temporarily overloaded across all users), a 429 rate_limit_error, or a 500 api_error. Those are the retryable status codes; the Anthropic SDKs already retry them with exponential backoff (twice by default, honoring the retry-after header). A fallback is the next layer: after retries are exhausted, route the same request down an alternate path instead of surfacing an error to the user.

🔑 Retry first (same path, wait), then fall back (different path). A fallback without a retry budget just hammers a second dependency; a retry without a fallback gives up once the primary stays down.

CONCEPT Four fallback tiers, cheapest degradation last

Order your fallbacks from "closest to the original answer" to "most degraded but still useful." Each rung trades a little quality or freshness for a much higher chance of returning something.

Fallback tierWhat changesWhen to useCost of the swap
Another model tierOpus → Sonnet → HaikuRepeated 529 / overload on the primary modelLower capability, cheaper & often less loaded
Another region / provider1st-party API → Bedrock / VertexWhole-provider outage or regional saturationFeature-parity gaps; separate quota pool
Cached / canned responseServe a prior or templated answerEverything upstream is downStale, but instant and free
Degraded modeSkip the LLM; return a rule-based / partial resultLast resort — keep the product usableNo generation, but no error page
Anthropic ships a server-side fallbacks parameter (beta) that re-runs a refused request on a fallback model inside the same call — one round trip, no client logic. For overload/region failover you own the routing client-side, but the mental model is identical: name the plan B up front.
⚠️ 529 is shared-capacity overload; a sudden 429 spike can instead be your org's acceleration limit — ramp traffic gradually rather than failing straight to a fallback on every 429.

DIAGRAM The fallback ladder

flowchart TD A["Model request"] --> P["Primary: Opus, 1st-party"] P -->|"200 OK"| DONE["🟢 Return answer"] P -->|"429 / 500 / 529
(retryable)"| R{"Retries left?
(backoff)"} R -->|"yes"| P R -->|"exhausted"| T2["🔵 Another model tier
Sonnet / Haiku"] T2 -->|"still failing"| T3["🟠 Another region / provider
Bedrock / Vertex"] T3 -->|"still failing"| T4["🟠 Cached / canned response"] T4 -->|"nothing cached"| T5["🔴 Degraded mode
rule-based / partial"] classDef ok fill:#d4edda,stroke:#28a745,color:#155724; classDef alt fill:#cce5ff,stroke:#4a90d9,color:#0a3a6a; classDef gate fill:#ffe8cc,stroke:#ff9f40,color:#9a4a0a; classDef bad fill:#f8d7da,stroke:#dc3545,color:#721c24; class DONE ok; class T2 alt; class T3 gate; class T4 gate; class T5 bad;

CODE A fallback ladder around the Messages API

Retryable errors (429/500/529) climb the ladder; a bad request (400) is not retryable and re-raises.

# Cheapest-degradation-last ladder. SDK already retries each call twice;
# this handles what to do AFTER a path is exhausted.
TIERS = [
    ("claude-opus-4-8",   primary_client),   # 1st-party, top tier
    ("claude-sonnet-5",   primary_client),   # cheaper, often less loaded
    ("claude-opus-4-8",   bedrock_client),   # different region / provider
]

def answer_with_fallback(messages):
    for model, client in TIERS:
        try:
            return client.messages.create(
                model=model, max_tokens=1024, messages=messages)
        except (anthropic.RateLimitError,           # 429
                anthropic.InternalServerError,      # 500
                anthropic.APIStatusError) as e:      # 529 overloaded
            if getattr(e, "status_code", None) == 400:
                raise                                # not retryable — real bug
            log(f"tier {model} failed: {e}; trying next")
    return cached_or_degraded(messages)              # canned answer, marked degraded

TRADE-OFFS Falling back vs. failing fast

✅ Layered fallbacks

  • Availability survives a model/provider outage
  • User sees an answer, not an error page
  • Cheaper tiers absorb overload spikes

⚠️ Fallback everywhere, always

  • Silent quality drops hide real incidents
  • Falling back on a 400 masks a genuine bug
  • Extra latency & cost per attempt; parity gaps across providers
Fall back on retryable failures (429 / 5xx / 529) only. Surface 400/401/403/404 — those are your bug, not the provider's capacity.

EXAM TRAPS

Trap: "Retry a 400/404 through the fallback ladder." → Only 429, 500, 529 (and connection/timeout) are retryable. A 400 is a malformed request — retrying or falling back just repeats the bug.
Trap: Treating 529 as fatal. → overloaded_error is temporary capacity pressure. Back off and retry, then drop to a cheaper tier or another region — don't error the whole request.
Trap: Falling back with no signal to the caller. → A degraded/cached answer should be marked as such (see 6.3), or you'll debug a "wrong answer" that was really a silent fallback.
Trap: Firing every 429 straight to a fallback. → A 429 spike can be an org acceleration limit — ramp traffic gradually and honor retry-after before escalating.

PRACTICE

Q1 · Your Opus calls start returning 529 during a traffic spike. What's the fallback order?

Retry with backoff first (the SDK does this, honoring retry-after). If it persists, drop to a cheaper/less-loaded model tier (Sonnet/Haiku), then another region or provider (Bedrock/Vertex), then a cached/canned response, and finally a degraded rule-based path — never a raw error.

Q2 · Why should a 400 error not trigger a fallback?

400 invalid_request_error is non-retryable — the request itself is malformed (bad schema, unsupported param). Falling back re-sends the same broken request down another path and hides the real bug. Retryable codes are 429, 500, 529 (plus connection/timeout errors).

Q3 · What's the difference between the SDK's built-in retry and a fallback?

Retry = same path, wait, try again (SDK does this ~2× with backoff for 429/5xx). Fallback = a different path (cheaper model, other region, cache, degraded mode) once retries are exhausted. Retry handles blips; fallback handles a sustained outage of the primary path.