cleandataengineering.aiAgent Harness, drawn simply
You are here: Tech stack › L9 · Determinism & replay

Agent Harness Tutorial · Tech stack

L9 · Determinism & replay

A past decision can be re-run and explained. Family: See · around every turn.

AZURE OPTIONS(nothing on Azure)· considered: no service does thisOUR CHOICEOur run record + cassettesagent_harness/cassetteConfig fingerprintOPEN-SOURCE OPTIONSVCR.py· considered: not usedOwn recorder✓ used: chosenWHO DOES THE WORKAzure gives 0% · we code once 85% · each agent team still does 15%
Dark boxes are what we run. Solid boxes are used or planned; dashed boxes were considered or dropped (crossed). The bar: grey is what Azure gives, dark is what we code once, hatched is what each agent team writes.
Why our choiceNo product records an agent's decisions in a replayable form, so this is almost entirely our code.
OptionWhere fromVerdictWhy
Our run record + cassettesours✓ chosenagent_harness/cassette
Config fingerprintours✓ chosen
(nothing on Azure)Azure· consideredno service does this
VCR.pyopen source· considerednot used
Own recorderopen source✓ usedchosen

How it works inside the harness: Offline.

Compared: How do we replay a run without live calls?

Own cassettes at the model…ourschosenVCR.py (vcrpy)open sourceto adoptPydantic AI TestModel / Fu…open sourcechosenLangfuse / LangSmith replayvendornot used
Dark: what we chose. Crossed and dashed: rejected or not for us. From the Series.
OptionFromVerdictWhat it recordsLevelOn a missStatus
Own cassettes at the model choke pointourschosenrequest fingerprint → responsemodel callsrun fails, never goes livebuilt; used in release gates
VCR.py (vcrpy)open sourceto adoptHTTP requests and responsesHTTP, for tool callsconfigurable record modeto adopt for tools
Pydantic AI TestModel / FunctionModelopen sourcechosennothing: scripted repliesmodel layern/aused in tests
Langfuse / LangSmith replayvendornot usedtraced promptsone model call, in a UIn/a: re-runs livenot used
DecisionWe record model exchanges ourselves at the one model choke point and refuse to replay under a different model, tool list or temperature, so a miss fails loudly. VCR.py is to be adopted for HTTP-level tool cassettes; trace-tool playgrounds re-run live, so they are not replay.

How it is done: the real code

Cut from the live repositories each time the site is built, so it always matches what runs.

class CassetteMiss(AgentFailure):
    """Nothing recorded answers this request.

    Raised rather than falling through to the provider. A replay that reaches
    the network is not a replay — it is an offline suite that silently costs
    money and stops being reproducible on the day it matters.
    """

    fault = Fault.MISCONFIGURED