Eval Harness Tutorial · Five patterns
decision_vs_should: decide when the rules say
The question: Was a person asked to approve a large payout?
Some moments need a decision: ask a person to approve, hand over to a human. You name the decision and list when it should happen. The check finds two mistakes:
- Under: it should have been made and wasn't (a ₹61,000 payout with no approval).
- Over: it was made with no rule calling for it (a needless hand-over).
Conditions you can use: user_says, tool_called,
result_of (what a tool returned), tool (an argument above a value),
failed_calls_at_least, consecutive_failed_at_least, and
all: to need several at once.
The rule, as you'd write it in YAML
payout_approval:
adapter: decision_vs_should
decision: awaiting_approval
when:
- all:
- {tool_called: request_payout}
- {result_of: get_claim, field: approved_amount, above: 25000}A conversation that passes
verdict | pass |
score | 1.00 |
reason | 2 of 2 held |
evaluator | payout_approval |
provider | ours |
A conversation that fails
verdict | fail |
score | 0.00 |
reason | 0 of 1 held; no awaiting_approval although request_payout was called and get_claim returned approved_amount = 61000 |
evaluator | payout_approval |
provider | ours |
findings | under: no awaiting_approval although request_payout was called and get_claim returned approved_amount = 61000 |
What it can't see
- Judged per window: only the steps since the previous such decision count, so failures before a person handed back don't count toward the next hand-over.
- If you list only some of the spec's triggers, set
check: under. Otherwise a hand-over for a reason you didn't list looks needless. - The agent must record its decisions. Standard traces have no field for this yet (see gaps).
Decided · decision_vs_should
- Is “under” and “over” the right way to talk about this to ops people, or would they say it differently?
Decided: Keepunderandoverin the YAML; reports say “missed” and “needless”.