cleandataengineering.aiEval Harness, drawn simply
You are here: Requirements

Eval Harness Tutorial · Requirements

Requirements: what must be true

A requirement is one thing that must be true of the agent, written down before anyone builds or checks it. Without requirements, a check is just an opinion. With them, every fail says which promise was broken.

Requirements come from two places:

  • The agent's spec: what this business needs. FNOL's spec says “an amount above ₹25,000 needs a claims handler” and gives it the id P-PAYOUT. These are domain-specific.
  • The standard (the AI Assurance Catalog): what every AI system of a kind must do, such as AAC-0110, “a claimed action is supported by that action's result”. These are domain-agnostic; see the catalog.
RequirementP-PAYOUT: “an amount above₹25,000 needs a claimshandler”tested byOpen: Rule (YAML)Rule (YAML)payout_approvalrequirements: [P-PAYOUT]the file ›usesOpen: CheckCheckdecision_vs_shouldhelps meet AAC-0078(the standard)the pattern ›givesResultfail: no approvalfor ₹61,000→ P-PAYOUT
Each link is data, not a comment: the rule names its requirement ids, and the check declares the standard's obligations it helps meet.

How the link is written

examples/claims-fnol.yaml (part)
spec: clean-ai-engineering/drafts/examples/motor-claims-fnol.aoas.yaml

evaluators:
  payout_approval:
    description: A person approves exactly the payouts over ₹25,000.
    requirements: [P-PAYOUT, issue_payout.authority.agent_when]
    adapter: decision_vs_should
    ...

Kept honest by a test. If a rule names a requirement id that isn't in the spec (a typo, or a requirement that was renamed), the test suite fails. The wording on these pages is looked up in the spec when they're built, so it can't drift.

Coverage: how many requirements a rule tests

AgentRequirements in its specTested by a ruleShare
Motor-claims FNOL agent42819%
Clothing shop support agent42614%
Why coverage is low, and why that's fine for now

Many requirements are about what the agent's systems refuse (the claims system won't register a claim on a lapsed policy), and are tested in the agent's own scenarios, not after the fact. Others need checks we haven't built (a date promised in a reply). Each untested requirement is listed by name on the agent's page, so the gap is known and owned.

Decided · Requirements
  1. Should every rule be required to name at least one requirement, so no rule exists without a reason?
    Decided: Yes, as a warning rather than a refusal until a team has a spec. FNOL and the clothing agent already comply.
  2. Which untested FNOL requirements matter most to check after the fact?
    Decided: P-PAYOUT-OWED (pay when asked; a status question issues nothing), P-OWN-WORDS (act only on a named claim), P-OWNERSHIP, and the “never a date” half of P-CLAIM-STATUS.

Look inside