Agent Harness Tutorial · Testing
Testing: how we know it works
Most of an agent is ordinary software: the routes, the tools, the contracts, the state, the retries, the loop's limits. Test all of that with exact assertions, with the model replaced by a scripted one. Only the model's own choices (which tool, what it says, whether it is grounded and safe) need scoring over many runs, in a simulated world, against a bar.
Goes inA code change
What happensExact tests for our code; scenarios in a simulated world for the model's choices; a gate that compares with the last release
Comes outA release that is provably no worse, or a blocked one
| Software part | Agentic part | |
|---|---|---|
| What it tests | our code: routes, tools, contracts, state, retries, loop limits | the model's choices: which tool, what it says, grounded, safe |
| How it judges | exact pass or fail | scored over many cases, against a threshold and the last release |
| The model | replaced by a scripted model or recorded calls | real models, in AgentTwin's simulated world |
| Tools | pytest, contract tests, spec coverage, load tests, failure drills | golden sets, AgentTwin, pydantic-evals, judges, red team |
Today, in numbers. The claims agent: 750 tests from 35 files (0 failures, 80 s), 29 of 29 scenarios, 64 of 64 owed spec statements, 16 of 16 layers. The reference agent: 2,179 tests from 106 files, 48 scenarios, 185 of 185 statements.
Want every detail? The long version is in the Series: testing at a glance.
Look inside
The test pyramidMany cheap exact tests at the bottom, a few scored ones at the top.Unit tests: table-driven, tied to the specOne table of cases per behaviour; every test names the spec statements it proves.Contract tests: every plug fits its socketEvery adapter for a port passes the same rows: scripted, Groq or APIM; Keycloak or Entra.Recorded model calls (cassettes)Record a real model call once; replay it forever, free and offline; refuse a mismatch.Scenarios in AgentTwin: a simulated worldTwin the world, not the agent: the real agent runs unchanged against a simulated claims system, clock and people.Golden setsSingle cases with known right answers, generated from the spec's boundaries; never edited to pass.Red team: attacks, judged by the worldGenerated injections, property tests that try to break it, planted notes; any success blocks the release.Failure drillsKill it, cut it, slow it, send it twice: does it stay correct?Performance and loadNothing built yet: the plan is Azure Load Testing at three times peak.The gates and CIEvery push: lint, types, architecture, 750 tests, four gates, then images. A release needs all four green.Coverage: which promise was provedLine coverage says which code ran. Spec coverage says which promise was proved.Testing in productionSynthetic canaries, canary releases, shadow mode, and misses turned into tests.