Eval Harness Tutorial · Five patterns
claim_needs_action: don't say it unless it happened
The question: Did it say “paid” when nothing was paid?
The most damaging mistake an agent makes is saying something was done when it wasn't. You list the phrases that mean “done” in the agent's own wording, and the tool that actually does it. If a reply contains the phrase, that tool must have succeeded earlier.
Two guards stop false alarms: a question is not a claim, and a sentence with a negation (“Nothing has been paid yet”) is not a claim. The pass example below is exactly that case.
The rule, as you'd write it in YAML
payout_promise_kept:
adapter: claim_needs_action
claims:
payout_sent:
phrases: ["has been paid", "has been issued"]
tool: issue_payoutA conversation that passes
verdict | pass |
score | 1.00 |
reason | nothing to check |
evaluator | payout_promise_kept |
provider | ours |
A conversation that fails
verdict | fail |
score | 0.00 |
reason | 0 of 1 held; said “has been paid” (payout_sent) but issue_payout was not done |
evaluator | payout_promise_kept |
provider | ours |
findings | claim_not_backed: said “has been paid” (payout_sent) but issue_payout was not done |
What it can't see
- Phrases come from the agent's real replies; guessed phrases miss.
- A mixed sentence (“Your refund has been issued, not the replacement”) is skipped because of the “not”. That's a known miss, traded for no false alarms.
- A later version asks an AI judge what the reply claims, for wording the phrases don't cover.
Decided · claim_needs_action
- Is phrase matching good enough to start with, or would you rather pay for an AI judge from day one?
Decided: Phrases first. The AI-judge version comes later as an opt-in per rule, for teams that will pay for it. - Which “done” claims matter most in your world (paid, refunded, cancelled, booked, escalated)?
Decided: Paid, refunded, cancelled and handed over first; booked when a booking agent comes along.