Eval Harness Tutorial · Five patterns
Five patterns for whole conversations
Most of the questions ops teams ask about an agent fit five shapes. Each shape is a pattern: plain code that reads a session, written once by us. A domain team fills it in with their own tool names and limits in YAML; they don't write code.
| Pattern | The kind of question it answers |
|---|---|
requires_before | Was the claim read before it was paid? |
argument_limit | Was a refund above ₹10,000 sent? |
claim_needs_action | Did it say “paid” when nothing was paid? |
decision_vs_should | Was a person asked to approve a large payout? |
consistency | Did it state an amount no system gave it? |
How every pattern answers
Each pattern counts how many things it checked and which failed. The score is the share that held (1.0 when nothing applied). Any failure fails the check, and each failure says what and at which step. No AI model is involved: the same session always gets the same verdict.
Each page below shows one passing and one failing conversation. Both were run through the real code when this page was built; the verdicts are its actual output.
Look inside
requires_before: do A before BWas the claim read before it was paid?argument_limit: a number must stay within a limitWas a refund above ₹10,000 sent?claim_needs_action: don't say it unless it happenedDid it say “paid” when nothing was paid?decision_vs_should: decide when the rules sayWas a person asked to approve a large payout?consistency: facts must come from a systemDid it state an amount no system gave it?