cleandataengineering.aiEval Harness, drawn simply
You are here: Five patterns › consistency

Eval Harness Tutorial · Five patterns

consistency: facts must come from a system

The question: Did it state an amount no system gave it?

Every fact of a named kind in a reply (a claim id, an amount) must appear in an earlier tool result, or in what the customer said. You name each kind of fact without writing a regular expression: a built-in type such as inr_amount, or a shape such as CLM-###### (# a digit, A a letter). Each type compares values its own way, so “Rs. 12,400”, “₹12,400” and 12400 in a result all match.

The rule, as you'd write it in YAML
facts_from_systems:
  adapter: consistency
  facts:
    claim_id: {shape: "CLM-######"}   # # a digit, A a letter
    amount: inr_amount              # ₹12,400 · Rs. 12,400 · INR 12400

A conversation that passes

CUSTOMERAGENTTOOLS (its systems)Customer“How much will I get forCLM-104233?”calls get_claimid=CLM-104233get_claim returnsid: CLM-104233, status: approved,approved_amount: 12400Agent says“Claim CLM-104233 is approved for₹12,400.”
Passes.
verdictpass
score1.00
reason2 of 2 held
evaluatorfacts_from_systems
providerours

A conversation that fails

CUSTOMERAGENTTOOLS (its systems)Customer“How much will I get forCLM-104233?”calls get_claimid=CLM-104233get_claim returnsid: CLM-104233, status: approved,approved_amount: 12400✗ Agent says“Claim CLM-104233 is approved for₹15,000.”✗ step 4: amount “₹15,000” is in no earlier tool result
Fails: the hatched step is where the check found the problem.
verdictfail
score0.50
reason1 of 2 held; amount “₹15,000” is in no earlier tool result
evaluatorfacts_from_systems
providerours
findingsunsupported_fact: amount “₹15,000” is in no earlier tool result

What it can't see

  • It catches invented facts, not wrong arithmetic done on true ones.
  • Repeating the customer's own claim id back is allowed; that's not inventing it.
Decided · consistency
  1. Which facts would an ops team most want guarded: amounts, ids, dates, policy terms?
    Decided: Amounts and ids now. Dates next, because they need one format before they can be compared.