Eval Harness Tutorial · Five patterns
requires_before: do A before B
The question: Was the claim read before it was paid?
Some actions are only safe after another one. Never pay a claim nobody
has read; never cancel an order you haven't looked up. You name the action
(issue_payout), the one that must come first (get_claim),
and, optionally, which arguments must match (same: [id]), so reading
claim A doesn't count as reading claim B.
The earlier call must also have succeeded: a lookup that failed doesn't count.
The rule, as you'd write it in YAML
payout_needs_claim_lookup: adapter: requires_before action: issue_payout before: get_claim same: [id]
A conversation that passes
verdict | pass |
score | 1.00 |
reason | 1 of 1 held |
evaluator | payout_needs_claim_lookup |
provider | ours |
A conversation that fails
verdict | fail |
score | 0.00 |
reason | 0 of 1 held; issue_payout without an earlier get_claim for the same id |
evaluator | payout_needs_claim_lookup |
provider | ours |
findings | missing_before: issue_payout without an earlier get_claim for the same id |
What it can't see
- It checks order, not content: it doesn't check that the claim said “approved”. For that, use
decision_vs_shouldwithresult_of. - The score is the share of
issue_payoutcalls that had a lookup first.
Decided · requires_before
- In your ops world, what are the top three “A before B” rules? Do they all fit this shape?
Decided: Starting set: pay only after reading the claim; cancel or refund only after reading the order; change account details only after verifying the customer. All three fit this shape.