Agent Harness Tutorial · End to end · Zoom level 3
Layers around the call
The call to the model is wrapped in layers, like an onion. Each layer does one small job, and the loop sees none of them.
| What went wrong | What the layer calls it | What happens next |
|---|---|---|
| 429: too many calls a minute | Throttled | Wait as long as told, then try again |
| Timeout, 5xx, can't connect | Unavailable | Wait 0.2 s, then 0.4 s, 0.8 s ... (at most 3 tries) |
| 429 with budget_exceeded | Budget exhausted | The turn fails; no retry |
| 403: APIM blocked the prompt | Refused | The turn ends: "I am having trouble answering right now." |
| The output is not valid | Malformed | Never retried; see reading the answer |
All retries in one turn share one allowance of 4, so a turn can't retry forever. After 3 failures in a row the breaker opens and calls fail fast for 10 seconds, which stops us hammering a provider that is already down.
There is no second model to fall back to today. Retries and the breaker are the whole plan.
Code view: input, process, output, the call stack, the classes
Goes inone ModelRequest and a flaky provider
What happensResilientLLM refuses fast while the breaker is open, waits as long as the provider asked when rate-limited, and retries other failures with backoff from one shared per-turn budget.
Comes outa response, or one typed ModelUnavailable
How it is done: the real code
Cut from the live repositories each time the site is built, so it always matches what runs.
agent_harness/llm/pydantic_ai.py:133Where the model's output becomes our typesdef from_response(response: pa.ModelResponse) -> ModelResponse:
"""The typed boundary (AHC-0001), on Pydantic AI's parsed response. The same
two outcomes as `_from_wire`: a response, or `ModelMalformed` for tool
arguments that are not a JSON object and for a paid-for answer with neither
text nor a call (F-031)."""
calls: list[ToolCall] = []agent_harness/resilience/__init__.py:88One retry allowance shared by the whole turn@contextmanager
def unit_retries(limit: int) -> Iterator[None]:
"""One unit of work's retry allowance, shared by every call inside it.
AHC-0024 and F-087: "a retry count exists per unit of work, not per call
site". Each call kept its own few attempts, so a twelve-step turn could
retry twenty-four times. Inside this scope every retry spends from one
budget; when it is gone a failure is raised instead of retried. Outside any
scope only the per-call bound applies.
"""
token = _UNIT.set([limit])
try:
yield
finally:
_UNIT.reset(token)agent_harness/resilience/__init__.py:250Retries and the circuit breakerclass ResilientLLM:
"""L2 · L10. Any model client, behind a declared degradation path.
Three controls, each answering a different condition — and the reason they
are one decorator is that they must agree about which condition this is:
- **A rate limit** (`ModelThrottled`) waits exactly the provider's
`retry-after`, shared across every caller, and is **not** a breaker
failure — being told to slow down is not evidence the provider is down
(AHC-0021).
- **A failure** (`ModelUnavailable`) is retried with capped, jittered backoff,
a bounded number of times, each retry a span (AHC-0024), and counts
towards the breaker.
- **An open breaker** fails fast without calling the provider at all, until a
single probe is allowed after the cooldown (AHC-0005).
Malformed output is none of these and is never retried: the same prompt to
the same model produced something unreadable, and a retry mostly buys a
second bill. When the attempts run out, the last provider failure is raised
typed, and the loop turns it into the customer's `Failed`.
"""
def __init__(
self,
inner: LLMClient,
*,
attempts: int = 3,
backoff: Backoff | None = None,
breaker: CircuitBreaker | None = None,
throttle: Throttle | None = None,
sleep: Callable[[float], Awaitable[None]] | None = None,
) -> None:
self.inner = inner
self.attempts = attemptsTech choices (what we chose, Azure and open-source options): L10 · Failure handling