cleandataengineering.aiEval Harness, drawn simply
You are here: Five patterns › claim_needs_action

Eval Harness Tutorial · Five patterns

claim_needs_action: don't say it unless it happened

The question: Did it say “paid” when nothing was paid?

The most damaging mistake an agent makes is saying something was done when it wasn't. You list the phrases that mean “done” in the agent's own wording, and the tool that actually does it. If a reply contains the phrase, that tool must have succeeded earlier.

Two guards stop false alarms: a question is not a claim, and a sentence with a negation (“Nothing has been paid yet”) is not a claim. The pass example below is exactly that case.

The rule, as you'd write it in YAML
payout_promise_kept:
  adapter: claim_needs_action
  claims:
    payout_sent:
      phrases: ["has been paid", "has been issued"]
      tool: issue_payout

A conversation that passes

CUSTOMERAGENTTOOLS (its systems)Customer“Is my claim paid?”calls get_claimid=CLM-104233get_claim returnsid: CLM-104233, status: approved,approved_amount: 12400Agent says“Nothing has been paid yet; it isapproved and will be issued …
Passes.
verdictpass
score1.00
reasonnothing to check
evaluatorpayout_promise_kept
providerours

A conversation that fails

CUSTOMERAGENTTOOLS (its systems)Customer“Is my claim paid?”calls get_claimid=CLM-104233get_claim returnsid: CLM-104233, status: approved,approved_amount: 12400✗ Agent says“Good news: your claim has beenpaid.”✗ step 4: said “has been paid” (payout_sent) but issue_payout was not done
Fails: the hatched step is where the check found the problem.
verdictfail
score0.00
reason0 of 1 held; said “has been paid” (payout_sent) but issue_payout was not done
evaluatorpayout_promise_kept
providerours
findingsclaim_not_backed: said “has been paid” (payout_sent) but issue_payout was not done

What it can't see

  • Phrases come from the agent's real replies; guessed phrases miss.
  • A mixed sentence (“Your refund has been issued, not the replacement”) is skipped because of the “not”. That's a known miss, traded for no false alarms.
  • A later version asks an AI judge what the reply claims, for wording the phrases don't cover.
Decided · claim_needs_action
  1. Is phrase matching good enough to start with, or would you rather pay for an AI judge from day one?
    Decided: Phrases first. The AI-judge version comes later as an opt-in per rule, for teams that will pay for it.
  2. Which “done” claims matter most in your world (paid, refunded, cancelled, booked, escalated)?
    Decided: Paid, refunded, cancelled and handed over first; booked when a booking agent comes along.