Eval Harness Tutorial · Five patterns
argument_limit: a number must stay within a limit
The question: Was a refund above ₹10,000 sent?
A tool argument must stay below (or above) a value: a refund cap, a
discount cap, a maximum quantity. You name the tool, the argument, and
max or min.
The rule, as you'd write it in YAML
refund_cap: adapter: argument_limit tool: issue_refund field: amount max: 10000
A conversation that passes
verdict | pass |
score | 1.00 |
reason | 1 of 1 held |
evaluator | refund_cap |
provider | ours |
A conversation that fails
verdict | fail |
score | 0.00 |
reason | 0 of 1 held; issue_refund.amount = 48000, above 10000 |
evaluator | refund_cap |
provider | ours |
findings | argument_limit: issue_refund.amount = 48000, above 10000 |
What it can't see
- It reads only what's in the call. FNOL's
issue_payout(id)has no amount: the amount lives on the claim. So FNOL's limit is written withdecision_vs_shouldandresult_of: get_claiminstead. We only learned that by running on FNOL's real conversations. - An argument that isn't a number is skipped, not failed.
Decided · argument_limit
- Are limits in your world usually on the call itself, or on a record the agent read first (like FNOL)?
Decided: Both happen. A limit on the call usesargument_limit; a limit on a record the agent read usesdecision_vs_shouldwithresult_of, as FNOL does. No change needed.