Eval Harness Tutorial · Five patterns
consistency: facts must come from a system
The question: Did it state an amount no system gave it?
Every fact of a named kind in a reply (a claim id, an amount) must
appear in an earlier tool result, or in what the customer said. You name each
kind of fact without writing a regular expression: a built-in type such as
inr_amount, or a shape such as CLM-###### (# a digit, A a letter).
Each type compares values its own way, so “Rs. 12,400”, “₹12,400” and 12400 in a result all match.
The rule, as you'd write it in YAML
facts_from_systems:
adapter: consistency
facts:
claim_id: {shape: "CLM-######"} # # a digit, A a letter
amount: inr_amount # ₹12,400 · Rs. 12,400 · INR 12400A conversation that passes
verdict | pass |
score | 1.00 |
reason | 2 of 2 held |
evaluator | facts_from_systems |
provider | ours |
A conversation that fails
verdict | fail |
score | 0.50 |
reason | 1 of 2 held; amount “₹15,000” is in no earlier tool result |
evaluator | facts_from_systems |
provider | ours |
findings | unsupported_fact: amount “₹15,000” is in no earlier tool result |
What it can't see
- It catches invented facts, not wrong arithmetic done on true ones.
- Repeating the customer's own claim id back is allowed; that's not inventing it.
Decided · consistency
- Which facts would an ops team most want guarded: amounts, ids, dates, policy terms?
Decided: Amounts and ids now. Dates next, because they need one format before they can be compared.