cleandataengineering.aiEval Harness, drawn simply
You are here: Five patterns › argument_limit

Eval Harness Tutorial · Five patterns

argument_limit: a number must stay within a limit

The question: Was a refund above ₹10,000 sent?

A tool argument must stay below (or above) a value: a refund cap, a discount cap, a maximum quantity. You name the tool, the argument, and max or min.

The rule, as you'd write it in YAML
refund_cap:
  adapter: argument_limit
  tool: issue_refund
  field: amount
  max: 10000

A conversation that passes

CUSTOMERAGENTTOOLS (its systems)Customer“Refund my jacket, order 881.”calls issue_refundorder_id=881, amount=4999issue_refund returnsrefunded: 4999Agent says“Your refund of ₹4,999 is on itsway.”
Passes.
verdictpass
score1.00
reason1 of 1 held
evaluatorrefund_cap
providerours

A conversation that fails

CUSTOMERAGENTTOOLS (its systems)Customer“Refund my sofa, order 902.”✗ calls issue_refundorder_id=902, amount=48000issue_refund returnsrefunded: 48000Agent says“Your refund of ₹48,000 is on itsway.”✗ step 2: issue_refund.amount = 48000, above 10000
Fails: the hatched step is where the check found the problem.
verdictfail
score0.00
reason0 of 1 held; issue_refund.amount = 48000, above 10000
evaluatorrefund_cap
providerours
findingsargument_limit: issue_refund.amount = 48000, above 10000

What it can't see

  • It reads only what's in the call. FNOL's issue_payout(id) has no amount: the amount lives on the claim. So FNOL's limit is written with decision_vs_should and result_of: get_claim instead. We only learned that by running on FNOL's real conversations.
  • An argument that isn't a number is skipped, not failed.
Decided · argument_limit
  1. Are limits in your world usually on the call itself, or on a record the agent read first (like FNOL)?
    Decided: Both happen. A limit on the call uses argument_limit; a limit on a record the agent read uses decision_vs_should with result_of, as FNOL does. No change needed.