D1 · Agentic Architecture › Cluster 6 · Reliability

6.2 · Circuit Breakers

Stop calling a failing dependency before it takes you down

CONCEPT What a circuit breaker does

A circuit breaker wraps a tool or sub-agent and watches its error rate. Once failures cross a threshold, the breaker trips and stops sending calls to that component for a cooldown — failing fast (or falling back) instead of hammering a dead dependency. This is the direct fix for the cascade failure mode from 6.1: it prevents wasted calls and stops one broken component from dragging down the rest.

🔑 A retry re-issues one call. A circuit breaker governs the aggregate — it decides whether the component is worth calling at all right now. They compose: retry within a call, breaker across calls.

DIAGRAM The three-state machine

stateDiagram-v2 [*] --> Closed Closed --> Open: error rate crosses threshold
(N failures) Open --> HalfOpen: cooldown elapsed HalfOpen --> Closed: trial call succeeds HalfOpen --> Open: trial call fails note right of Closed: calls flow through normally note right of Open: calls are blocked / fall back
fail fast, no wasted calls note right of HalfOpen: let ONE trial call through
StateBehaviourTransition out
ClosedNormal — calls pass through; failures are counted→ Open after N failures (threshold crossed)
OpenTripped — calls are blocked and fail fast / fall back immediately→ Half-open once the cooldown timer elapses
Half-openTrial — let a single probe call throughSuccess → Closed; failure → back to Open

CODE A minimal breaker wrapper

Wrap any callable — an SDK call or a tool. The breaker short-circuits when open, probes when half-open, and resets on a successful probe.

import time, anthropic

class CircuitBreaker:
    def __init__(self, threshold=5, cooldown=30):
        self.threshold, self.cooldown = threshold, cooldown
        self.failures = 0
        self.state = "closed"
        self.opened_at = 0.0

    def call(self, fn, *args, **kw):
        if self.state == "open":
            if time.monotonic() - self.opened_at < self.cooldown:
                raise RuntimeError("circuit open — failing fast")  # no wasted call
            self.state = "half-open"                             # cooldown elapsed → probe
        try:
            result = fn(*args, **kw)
        except Exception:
            self._on_failure()
            raise
        self._on_success()                                    # closed / probe passed
        return result

    def _on_failure(self):
        self.failures += 1
        if self.state == "half-open" or self.failures >= self.threshold:
            self.state, self.opened_at = "open", time.monotonic()  # trip

    def _on_success(self):
        self.failures, self.state = 0, "closed"            # reset

# Usage: guard a flaky tool so its outages can't cascade.
breaker = CircuitBreaker(threshold=5, cooldown=30)
def run_search_tool(query):
    return breaker.call(flaky_search_api, query)
⚠️ Count only failures the breaker can act on. A 400 invalid_request is your bug and will fail identically on retry — tripping on it just hides the real error. Trip on transient/dependency failures (5xx, 529, connection, timeouts), not on 4xx client errors (see 6.3).

TRADE-OFFS Sensitivity of the threshold

✅ Trip early (low threshold / short cooldown)

  • Stops cascades fast; sheds load quickly
  • Recovers responsiveness for the rest of the system

⚠️ Trip early — the cost

  • A brief blip can open the circuit and block healthy traffic
  • Too-short cooldown flaps between open/half-open
  • May trigger fallback/degradation more often than needed
Tune to the dependency: rare-but-real outages want a higher threshold; a truly critical, frequently-flaky dependency wants an early trip plus a defined fallback (6.4).

EXAM TRAPS

Trap: "Add retries to fix the cascade." → Retries against a dead dependency accelerate the cascade. The correct primitive is a circuit breaker that stops calling it.
Trap: Confusing the states. → Closed = healthy (calls flow). Open = tripped (calls blocked). It's the inverse of an electrical switch, which trips people up.
Trap: Going open → closed directly. → You must pass through half-open (a single trial call) first; jumping straight to closed risks re-opening the floodgates onto a still-broken dependency.

PRACTICE

Q1 · Which state lets exactly one probe call through, and what happens on each outcome?

Half-open. It permits a single trial call. Success → transition to closed (resume normal traffic); failure → back to open and restart the cooldown.

Q2 · Should a circuit breaker count HTTP 400 errors toward its threshold?

No. A 400 is a client-side invalid_request that will fail identically every time — it's not a dependency outage. Tripping on it masks a bug you should fix. Count transient/dependency failures (5xx, 529, connection errors, timeouts).

Q3 · Breaker and retry are both configured on the same tool. How do they interact?

They compose at different scopes. Retry handles a single transient call (rung 1). The breaker tracks the aggregate error rate across calls and, once tripped, stops even attempting retries — which is what prevents retry storms from cascading.