Agent Harness Tutorial · Tech stack
L9 · Determinism & replay
A past decision can be re-run and explained. Family: See · around every turn.
Why our choiceNo product records an agent's decisions in a replayable form, so this is almost entirely our code.
| Option | Where from | Verdict | Why |
|---|---|---|---|
| Our run record + cassettes | ours | ✓ chosen | agent_harness/cassette |
| Config fingerprint | ours | ✓ chosen | |
| (nothing on Azure) | Azure | · considered | no service does this |
| VCR.py | open source | · considered | not used |
| Own recorder | open source | ✓ used | chosen |
How it works inside the harness: Offline.
Compared: How do we replay a run without live calls?
| Option | From | Verdict | What it records | Level | On a miss | Status |
|---|---|---|---|---|---|---|
| Own cassettes at the model choke point | ours | chosen | request fingerprint → response | model calls | run fails, never goes live | built; used in release gates |
| VCR.py (vcrpy) | open source | to adopt | HTTP requests and responses | HTTP, for tool calls | configurable record mode | to adopt for tools |
| Pydantic AI TestModel / FunctionModel | open source | chosen | nothing: scripted replies | model layer | n/a | used in tests |
| Langfuse / LangSmith replay | vendor | not used | traced prompts | one model call, in a UI | n/a: re-runs live | not used |
DecisionWe record model exchanges ourselves at the one model choke point and refuse to replay under a different model, tool list or temperature, so a miss fails loudly. VCR.py is to be adopted for HTTP-level tool cassettes; trace-tool playgrounds re-run live, so they are not replay.
How it is done: the real code
Cut from the live repositories each time the site is built, so it always matches what runs.
class CassetteMiss(AgentFailure):
"""Nothing recorded answers this request.
Raised rather than falling through to the provider. A replay that reaches
the network is not a replay — it is an offline suite that silently costs
money and stops being reproducible on the day it matters.
"""
fault = Fault.MISCONFIGURED