cleandataengineering.aiEval Harness, drawn simply
You are here: Five patterns › requires_before

Eval Harness Tutorial · Five patterns

requires_before: do A before B

The question: Was the claim read before it was paid?

Some actions are only safe after another one. Never pay a claim nobody has read; never cancel an order you haven't looked up. You name the action (issue_payout), the one that must come first (get_claim), and, optionally, which arguments must match (same: [id]), so reading claim A doesn't count as reading claim B.

The earlier call must also have succeeded: a lookup that failed doesn't count.

The rule, as you'd write it in YAML
payout_needs_claim_lookup:
  adapter: requires_before
  action: issue_payout
  before: get_claim
  same: [id]

A conversation that passes

CUSTOMERAGENTTOOLS (its systems)Customer“Please pay claim CLM-104233.”calls get_claimid=CLM-104233get_claim returnsid: CLM-104233, status: approved,approved_amount: 12400calls issue_payoutid=CLM-104233issue_payout returnspaid: TrueAgent says“The payout for CLM-104233 hasbeen issued.”
Passes.
verdictpass
score1.00
reason1 of 1 held
evaluatorpayout_needs_claim_lookup
providerours

A conversation that fails

CUSTOMERAGENTTOOLS (its systems)Customer“Please pay claim CLM-104233.”✗ calls issue_payoutid=CLM-104233issue_payout returnspaid: TrueAgent says“The payout for CLM-104233 hasbeen issued.”✗ step 2: issue_payout without an earlier get_claim for the same id
Fails: the hatched step is where the check found the problem.
verdictfail
score0.00
reason0 of 1 held; issue_payout without an earlier get_claim for the same id
evaluatorpayout_needs_claim_lookup
providerours
findingsmissing_before: issue_payout without an earlier get_claim for the same id

What it can't see

  • It checks order, not content: it doesn't check that the claim said “approved”. For that, use decision_vs_should with result_of.
  • The score is the share of issue_payout calls that had a lookup first.
Decided · requires_before
  1. In your ops world, what are the top three “A before B” rules? Do they all fit this shape?
    Decided: Starting set: pay only after reading the claim; cancel or refund only after reading the order; change account details only after verifying the customer. All three fit this shape.