cleandataengineering.aiAgent Harness, drawn simply
You are here: Testing

Agent Harness Tutorial · Testing

Testing: how we know it works

Most of an agent is ordinary software: the routes, the tools, the contracts, the state, the retries, the loop's limits. Test all of that with exact assertions, with the model replaced by a scripted one. Only the model's own choices (which tool, what it says, whether it is grounded and safe) need scoring over many runs, in a simulated world, against a bar.

Goes inA code change
What happensExact tests for our code; scenarios in a simulated world for the model's choices; a gate that compares with the last release
Comes outA release that is provably no worse, or a blocked one
Click any box to open its page.
Software partAgentic part
What it testsour code: routes, tools, contracts, state, retries, loop limitsthe model's choices: which tool, what it says, grounded, safe
How it judgesexact pass or failscored over many cases, against a threshold and the last release
The modelreplaced by a scripted model or recorded callsreal models, in AgentTwin's simulated world
Toolspytest, contract tests, spec coverage, load tests, failure drillsgolden sets, AgentTwin, pydantic-evals, judges, red team

Today, in numbers. The claims agent: 750 tests from 35 files (0 failures, 80 s), 29 of 29 scenarios, 64 of 64 owed spec statements, 16 of 16 layers. The reference agent: 2,179 tests from 106 files, 48 scenarios, 185 of 185 statements.

Want every detail? The long version is in the Series: testing at a glance.

Look inside