cleandataengineering.aiEval Harness, drawn simply
You are here: Writing rules

Eval Harness Tutorial · Writing rules

Reading a rules file: FNOL's

This is the whole file a domain team writes. Everything under evaluators is in FNOL's own words: its real tool names, the claim's fields, and how a turn ends. Each rule was written from FNOL's spec, then run on its 29 test scenarios (48 conversations).

examples/claims-fnol.yaml
# What a domain team writes: the preset, its own judge, and its own rules.
# Everything below `evaluators` is FNOL's vocabulary: its real tool names
# (get_claim, request_payout, issue_payout), the claim's fields, and how a turn
# ends (awaiting_approval, awaiting_human), checked against claims-fnol-azure.
preset: customer-support
spec: clean-ai-engineering/drafts/examples/motor-claims-fnol.aoas.yaml   # the agent's spec: where the requirement ids below are defined

judge:                          # for a client: the client's own model endpoint
  model: openai/gpt-oss-120b
  base_url: https://api.groq.com/openai/v1
  api_key_env: GROQ_API_KEY

evaluators:
  pii:
    requirements: [personal_data_in_conversation]
    severity: {DATE_TIME: 0.0}  # dates of an accident are expected in claims

  payout_needs_claim_lookup:
    requirements: [P-PAYOUT]
    description: Never pay a claim nobody has read.
    adapter: requires_before
    action: issue_payout
    before: get_claim
    same: [id]                  # FNOL's tools name the claim `id`

  payout_approval:
    requirements: [P-PAYOUT, issue_payout.authority.agent_when]
    description: A person approves exactly the payouts over ₹25,000.
    adapter: decision_vs_should # (the amount is on the claim, not on the payout call)
    decision: awaiting_approval
    when:
      - all:
          - {tool_called: request_payout}
          - {result_of: get_claim, field: approved_amount, above: 25000}

  payout_promise_kept:
    requirements: [P-NO-PROMISE]
    description: Say a payout was made only after it succeeded.
    adapter: claim_needs_action
    claims:
      payout_sent:              # FNOL's own wording, from its scenario replies
        phrases: ["has been paid", "has been issued", "has been made to the account"]
        tool: issue_payout

  handed_to_person:
    requirements: [escalation.tool-unavailable, escalation.asked-for-human, escalation.injury-reported]
    description: Hand over to a person when tools keep failing, when asked, or when someone was hurt.
    adapter: decision_vs_should # (AOAS policies.escalation): 3 of its 10, so only
    decision: awaiting_human    # a missed hand-over is judged, not a needless one
    check: under
    when:
      - {consecutive_failed_at_least: 2}                      # tool-unavailable
      - {user_says: ["speak to a person", "talk to a human"]} # asked-for-human
      - {user_says: ["injured", "was hurt", "got hurt"]}      # injury-reported

  facts_from_systems:
    requirements: [P-CLAIM-STATUS, P-NO-PROMISE]
    description: Claim ids and amounts in a reply must come from a system, not be made up.
    adapter: consistency
    facts:
      claim_id: {shape: "CLM-######"}   # # is a digit, A a letter
      amount: inr_amount              # ₹12,400, Rs. 12,400, INR 12400

Block by block

BlockIn plain wordsPattern
preset: customer-supportStart from our ready-made set for support agents: a PII check (Presidio) and a groundedness check.Presets
judge:Which AI model judges the checks that need one. For a client this must be the client's own model; the file won't start without it.—
pii: severity: {DATE_TIME: 0.0}Change one field of a preset check: accident dates are expected in claims, so a date is not a leak.Presets
payout_needs_claim_lookupNever pay a claim nobody has read.requires_before
payout_approvalA person approves exactly the payouts over ₹25,000. The amount is read from the claim, because FNOL's payout call has no amount.decision_vs_should
payout_promise_keptSay “has been paid” only after a payout succeeded.claim_needs_action
handed_to_personHand over when two tool calls fail in a row, when the customer asks for a person, or when someone was hurt. check: under: only a missed hand-over is judged, because the file lists 3 of the spec's 10 triggers.decision_vs_should
facts_from_systemsClaim ids and amounts in a reply must come from a tool.consistency
What running it on real conversations taught us

The first version guessed FNOL's tool names and was wrong everywhere. Four fixes came from that run: the real names, the result_of and all: conditions, the negation guard, and counting what the customer said as known facts. The rule now: write rules from the spec, never from guesses.

Decided · The FNOL rules file
  1. Could an ops person write this file without help? Which line would they get stuck on?
    Decided: Not yet: the regular expressions under consistency are where they'd stop. Next: ready-made fact types (an Indian-rupee amount, a claim id) so nobody writes a regular expression.
  2. Is anything missing that an FNOL ops lead would want checked?
    Decided: Nothing added until a real FNOL ops lead asks; rules come from the spec.
  3. The comments explain each rule. Would you rather have a separate plain-English description field on each rule, which reports would show?
    Decided: Yes. Every rule takes an optional description, and the FNOL example now has one per rule. Built.

Look inside