cleandataengineering.aiEval Harness, drawn simply
You are here: Start here › Why check?

Eval Harness Tutorial · Start here

Why check an agent at all?

Mistakes come in three sizes, and each needs a different kind of check.

1 · One replyDoes this answer leak aphone number? Is it basedon the documents?2 · One conversationDid it look the claim upbefore paying it? Did itdo what it said it did?3 · Many conversationsIs cost per conversationcreeping up? Same questionasked 500 times?provider checks (Presidio, Azure…)our five patterns ← this tutorialcost rules (not built yet)
Most tools on the market check size 1. Size 2 is where ops teams' real questions live, and it's what the five patterns are for.
  1. One reply. You can judge it by looking at that reply alone. Many tools already do this well; we plug them in rather than rewrite them.
  2. One conversation. You can't judge a payout from the reply. You need the order of events: was the claim read first, did the payout succeed, was a person asked when the amount was large. These are questions like “a cancel happened with no order lookup before it”. They need the whole conversation, in order. That's a session.
  3. Many conversations. Some problems only show across days or thousands of calls, such as cost creeping up. These run on a separate engine later.

Two more ideas run through everything:

  • Write the rules from the agent's spec, never from guesses. When we guessed FNOL's tool names, every check was wrong. When we wrote them from the spec, they were right.
  • Same answer every run. The five patterns are plain code, with no AI model judging. A rule that says “the payout needs a lookup first” gives the same verdict every time, so you can trust a fail.