Eval Harness Tutorial · Start here
Why check an agent at all?
Mistakes come in three sizes, and each needs a different kind of check.
- One reply. You can judge it by looking at that reply alone. Many tools already do this well; we plug them in rather than rewrite them.
- One conversation. You can't judge a payout from the reply. You need the order of events: was the claim read first, did the payout succeed, was a person asked when the amount was large. These are questions like “a cancel happened with no order lookup before it”. They need the whole conversation, in order. That's a session.
- Many conversations. Some problems only show across days or thousands of calls, such as cost creeping up. These run on a separate engine later.
Two more ideas run through everything:
- Write the rules from the agent's spec, never from guesses. When we guessed FNOL's tool names, every check was wrong. When we wrote them from the spec, they were right.
- Same answer every run. The five patterns are plain code, with no AI model judging. A rule that says “the payout needs a lookup first” gives the same verdict every time, so you can trust a fail.