cleandataengineering.aiAgent Harness, drawn simply
You are here: End to end › Core loop › Model call › Layers around the call

Agent Harness Tutorial · End to end · Zoom level 3

Layers around the call

The call to the model is wrapped in layers, like an onion. Each layer does one small job, and the loop sees none of them.

1. Tidy the text: curly quotes and dashes made plain2. Count: calls and tokens, for the trace3. Retry and breaker: 3 tries; 3 failures pause it for 10 s4. Map to our types: Pydantic AI, one request5. HTTP client: 60 s timeout, no retries of its ownOpen: → APIM → Groq→ APIM → Groqzoom in ›
The request goes in from the outside; the response comes back out through the same layers.
What went wrongWhat the layer calls itWhat happens next
429: too many calls a minuteThrottledWait as long as told, then try again
Timeout, 5xx, can't connectUnavailableWait 0.2 s, then 0.4 s, 0.8 s ... (at most 3 tries)
429 with budget_exceededBudget exhaustedThe turn fails; no retry
403: APIM blocked the promptRefusedThe turn ends: "I am having trouble answering right now."
The output is not validMalformedNever retried; see reading the answer

All retries in one turn share one allowance of 4, so a turn can't retry forever. After 3 failures in a row the breaker opens and calls fail fast for 10 seconds, which stops us hammering a provider that is already down.

There is no second model to fall back to today. Retries and the breaker are the whole plan.

Code view: input, process, output, the call stack, the classes

Goes inone ModelRequest and a flaky provider
What happensResilientLLM refuses fast while the breaker is open, waits as long as the provider asked when rate-limited, and retries other failures with backoff from one shared per-turn budget.
Comes outa response, or one typed ModelUnavailable
CALL STACK · START TO FINISHGOES IN → COMES OUT1Open the source: agent_harness/loop/__init__.py:95run(...) # with unit_retries(...)agent_harness/loop/__init__.py:95in: budgets.max_retries_per_unitout: a per-turn retry allowance in a ContextVar2Open the source: agent_harness/resilience/__init__.py:89unit_retries(limit)agent_harness/resilience/__init__.py:89in: the limit, e.g. 4out: the _UNIT ContextVar set to [limit]3Open the source: agent_harness/resilience/__init__.py:289ResilientLLM.complete(request)agent_harness/resilience/__init__.py:289in: ModelRequestout: response or a typed failure4Open the source: agent_harness/resilience/__init__.py:192CircuitBreaker.closed # propertyagent_harness/resilience/__init__.py:192in: breaker stateout: False when open: raise ModelUnavailable5Open the source: agent_harness/resilience/__init__.py:176CircuitBreaker.state # propertyagent_harness/resilience/__init__.py:176in: opened time and cooldownout: CLOSED, OPEN or HALF_OPEN6Open the source: agent_harness/resilience/__init__.py:116with_retry(work, attempts=3, backoff=, retry_on=(_Retry,))agent_harness/resilience/__init__.py:116in: the attempt functionout: a response, or re-raise of the last _Retry7Open the source: agent_harness/resilience/__init__.py:78Backoff.delay(attempt)agent_harness/resilience/__init__.py:78in: attempt numberout: capped exponential delay with jitter8Open the source: agent_harness/resilience/__init__.py:308ResilientLLM._attempt(request, tries)agent_harness/resilience/__init__.py:308in: requestout: response, or raise _Retry / typed failure9Open the source: agent_harness/resilience/__init__.py:235Throttle.wait(sleep)agent_harness/resilience/__init__.py:235in: shared retry-after deadlineout: sleeps until it passes10Open the source: agent_harness/llm/pydantic_ai.py:179failure_of(error) # inside inner.completeagent_harness/llm/pydantic_ai.py:179in: a Pydantic AI exceptionout: Refused, BudgetExhausted, Throttled, Unavailable, Malformed11Open the source: agent_harness/resilience/__init__.py:221Throttle.note_retry_after(exc.retry_after)agent_harness/resilience/__init__.py:221in: a 429's retry-after secondsout: the shared wait deadline moved later12Open the source: agent_harness/resilience/__init__.py:185CircuitBreaker.record_failure()agent_harness/resilience/__init__.py:185in: a ModelUnavailableout: failure count up; OPEN at the threshold13Open the source: agent_harness/resilience/__init__.py:105_may_retry()agent_harness/resilience/__init__.py:105in: the unit's remaining allowanceout: True and one spent, or False (raise now)14Open the source: agent_harness/resilience/__init__.py:333ResilientLLM._visible_retry(reason, tried)agent_harness/resilience/__init__.py:333in: 'throttled' or 'unavailable'out: an agent.llm.retry span15Open the source: agent_harness/resilience/__init__.py:181CircuitBreaker.record_success()agent_harness/resilience/__init__.py:181in: a good responseout: failures reset, state CLOSED
Read top to bottom: each box is one function call, indented under the function that called it; the dark box is where it starts. Click a box to open its source file.
Dashed boxes are protocols (the ports) or enums; grey boxes are data models; white boxes are classes with behaviour. A dashed arrow means "implements". Click a class to open its source.

How it is done: the real code

Cut from the live repositories each time the site is built, so it always matches what runs.

agent_harness/llm/pydantic_ai.py:133Where the model's output becomes our types
def from_response(response: pa.ModelResponse) -> ModelResponse:
    """The typed boundary (AHC-0001), on Pydantic AI's parsed response. The same
    two outcomes as `_from_wire`: a response, or `ModelMalformed` for tool
    arguments that are not a JSON object and for a paid-for answer with neither
    text nor a call (F-031)."""
    calls: list[ToolCall] = []
agent_harness/resilience/__init__.py:88One retry allowance shared by the whole turn
@contextmanager
def unit_retries(limit: int) -> Iterator[None]:
    """One unit of work's retry allowance, shared by every call inside it.

    AHC-0024 and F-087: "a retry count exists per unit of work, not per call
    site". Each call kept its own few attempts, so a twelve-step turn could
    retry twenty-four times. Inside this scope every retry spends from one
    budget; when it is gone a failure is raised instead of retried. Outside any
    scope only the per-call bound applies.
    """
    token = _UNIT.set([limit])
    try:
        yield
    finally:
        _UNIT.reset(token)
agent_harness/resilience/__init__.py:250Retries and the circuit breaker
class ResilientLLM:
    """L2 · L10. Any model client, behind a declared degradation path.

    Three controls, each answering a different condition — and the reason they
    are one decorator is that they must agree about which condition this is:

    - **A rate limit** (`ModelThrottled`) waits exactly the provider's
      `retry-after`, shared across every caller, and is **not** a breaker
      failure — being told to slow down is not evidence the provider is down
      (AHC-0021).
    - **A failure** (`ModelUnavailable`) is retried with capped, jittered backoff,
      a bounded number of times, each retry a span (AHC-0024), and counts
      towards the breaker.
    - **An open breaker** fails fast without calling the provider at all, until a
      single probe is allowed after the cooldown (AHC-0005).

    Malformed output is none of these and is never retried: the same prompt to
    the same model produced something unreadable, and a retry mostly buys a
    second bill. When the attempts run out, the last provider failure is raised
    typed, and the loop turns it into the customer's `Failed`.
    """

    def __init__(
        self,
        inner: LLMClient,
        *,
        attempts: int = 3,
        backoff: Backoff | None = None,
        breaker: CircuitBreaker | None = None,
        throttle: Throttle | None = None,
        sleep: Callable[[float], Awaitable[None]] | None = None,
    ) -> None:
        self.inner = inner
        self.attempts = attempts

Tech choices (what we chose, Azure and open-source options): L10 · Failure handling