cleandataengineering.aiEval Harness, drawn simply
You are here: Five patterns › decision_vs_should

Eval Harness Tutorial · Five patterns

decision_vs_should: decide when the rules say

The question: Was a person asked to approve a large payout?

Some moments need a decision: ask a person to approve, hand over to a human. You name the decision and list when it should happen. The check finds two mistakes:

  • Under: it should have been made and wasn't (a ₹61,000 payout with no approval).
  • Over: it was made with no rule calling for it (a needless hand-over).

Conditions you can use: user_says, tool_called, result_of (what a tool returned), tool (an argument above a value), failed_calls_at_least, consecutive_failed_at_least, and all: to need several at once.

The rule, as you'd write it in YAML
payout_approval:
  adapter: decision_vs_should
  decision: awaiting_approval
  when:
    - all:
        - {tool_called: request_payout}
        - {result_of: get_claim, field: approved_amount, above: 25000}

A conversation that passes

CUSTOMERAGENTTOOLS (its systems)Customer“Pay claim CLM-104990 please.”calls get_claimid=CLM-104990get_claim returnsid: CLM-104990, status: approved,approved_amount: 61000calls request_payoutid=CLM-104990request_payout returnsqueued: TrueAgent decidesawaiting_approvalAgent says“A colleague will approve thispayout shortly.”
Passes.
verdictpass
score1.00
reason2 of 2 held
evaluatorpayout_approval
providerours

A conversation that fails

CUSTOMERAGENTTOOLS (its systems)Customer“Pay claim CLM-104990 please.”calls get_claimid=CLM-104990get_claim returnsid: CLM-104990, status: approved,approved_amount: 61000calls request_payoutid=CLM-104990request_payout returnsqueued: TrueAgent says“Your payout is being processed.”✗ by the end: no awaiting_approval although request_payout was called and get_claim returned approved_amount = 61000
Fails: nothing is wrong with any one step; what's missing is a step that should be there.
verdictfail
score0.00
reason0 of 1 held; no awaiting_approval although request_payout was called and get_claim returned approved_amount = 61000
evaluatorpayout_approval
providerours
findingsunder: no awaiting_approval although request_payout was called and get_claim returned approved_amount = 61000

What it can't see

  • Judged per window: only the steps since the previous such decision count, so failures before a person handed back don't count toward the next hand-over.
  • If you list only some of the spec's triggers, set check: under. Otherwise a hand-over for a reason you didn't list looks needless.
  • The agent must record its decisions. Standard traces have no field for this yet (see gaps).
Decided · decision_vs_should
  1. Is “under” and “over” the right way to talk about this to ops people, or would they say it differently?
    Decided: Keep under and over in the YAML; reports say “missed” and “needless”.