cleandataengineering.aiAgent Harness, drawn simply
You are here: Tech stack › L10 · Failure handling

Agent Harness Tutorial · Tech stack

L10 · Failure handling

Typed failures, fallbacks, undo paths, honest messages. Family: Control · around every turn.

AZURE OPTIONSAPIM backend pools + breaker◐ planned: tier 10Container Apps health probes◐ planned: in the designOUR CHOICEResilientLLMone retry budget per turnTyped failuresmapped from Pydantic AILedger abandon-and-retryOPEN-SOURCE OPTIONStenacity· considered: not usedOwn resilience wrapper✓ used: chosenWHO DOES THE WORKAzure gives 25% · we code once 55% · each agent team still does 20%
Dark boxes are what we run. Solid boxes are used or planned; dashed boxes were considered or dropped (crossed). The bar: grey is what Azure gives, dark is what we code once, hatched is what each agent team writes.
Why our choiceA retry hidden inside an SDK is invisible to our retry budget, so retries and fallbacks live in one place we control.
OptionWhere fromVerdictWhy
ResilientLLMours✓ chosenone retry budget per turn
Typed failuresours✓ chosenmapped from Pydantic AI
Ledger abandon-and-retryours✓ chosen
APIM backend pools + breakerAzure◐ plannedtier 10
Container Apps health probesAzure◐ plannedin the design
tenacityopen source· considerednot used
Own resilience wrapperopen source✓ usedchosen

How it works inside the harness: Layers around the call.

Compared: What retries and fails over when a call breaks?

ResilientLLM (own…ourschosentenacityopen sourcealternativeProvider SDK retr…vendorrejectedPydantic AI fallb…open sourcechosenAPIM backend pool…AzureplannedHealth probes + F…Azurechosen
Dark: what we chose. Crossed and dashed: rejected or not for us. Added here (not a Series table).
OptionFromVerdictWhere it actsKnows our typed failuresShares one retry budgetStatus
ResilientLLM (own wrapper)ourschoseninside the LLMClient portyesyesbuilt
tenacityopen sourcealternativeany Python callonly what we tell itif we wire itlisted open source option
Provider SDK retriesvendorrejectedinside the openai SDKnono: invisible to usoff: max_retries=0
Pydantic AI fallback modelopen sourcechoseninside the model adapterpartlyyes, one budgetdesign: degradation path
APIM backend pools + breakerAzureplannedat the gatewaynomust share our budgetplanned, Tier 10
Health probes + Front DoorAzurechosencontainers and regionsnon/afinal (design)
DecisionModel retries live in our ResilientLLM wrapper with one retry budget, so SDK retries are switched off to stay visible. APIM backend pools and the circuit breaker are planned for production failover across backends and regions.

How it is done: the real code

Cut from the live repositories each time the site is built, so it always matches what runs.

class ResilientLLM:
    """L2 · L10. Any model client, behind a declared degradation path.

    Three controls, each answering a different condition — and the reason they
    are one decorator is that they must agree about which condition this is:

    - **A rate limit** (`ModelThrottled`) waits exactly the provider's
      `retry-after`, shared across every caller, and is **not** a breaker
      failure — being told to slow down is not evidence the provider is down
      (AHC-0021).
    - **A failure** (`ModelUnavailable`) is retried with capped, jittered backoff,
      a bounded number of times, each retry a span (AHC-0024), and counts
      towards the breaker.
    - **An open breaker** fails fast without calling the provider at all, until a
      single probe is allowed after the cooldown (AHC-0005).

    Malformed output is none of these and is never retried: the same prompt to
    the same model produced something unreadable, and a retry mostly buys a
    second bill. When the attempts run out, the last provider failure is raised
    typed, and the loop turns it into the customer's `Failed`.
    """

    def __init__(
        self,
        inner: LLMClient,
        *,
        attempts: int = 3,
        backoff: Backoff | None = None,
        breaker: CircuitBreaker | None = None,
        throttle: Throttle | None = None,
        sleep: Callable[[float], Awaitable[None]] | None = None,
    ) -> None:
        self.inner = inner
        self.attempts = attempts