Eval Harness Tutorial · Requirements
Requirements: what must be true
A requirement is one thing that must be true of the agent, written down before anyone builds or checks it. Without requirements, a check is just an opinion. With them, every fail says which promise was broken.
Requirements come from two places:
- The agent's spec: what this business needs. FNOL's spec says “an amount
above ₹25,000 needs a claims handler” and gives it the id
P-PAYOUT. These are domain-specific. - The standard (the AI Assurance Catalog): what every AI system of a kind
must do, such as
AAC-0110, “a claimed action is supported by that action's result”. These are domain-agnostic; see the catalog.
How the link is written
spec: clean-ai-engineering/drafts/examples/motor-claims-fnol.aoas.yaml
evaluators:
payout_approval:
description: A person approves exactly the payouts over ₹25,000.
requirements: [P-PAYOUT, issue_payout.authority.agent_when]
adapter: decision_vs_should
...Kept honest by a test. If a rule names a requirement id that isn't in the spec (a typo, or a requirement that was renamed), the test suite fails. The wording on these pages is looked up in the spec when they're built, so it can't drift.
Coverage: how many requirements a rule tests
| Agent | Requirements in its spec | Tested by a rule | Share |
|---|---|---|---|
| Motor-claims FNOL agent | 42 | 8 | 19% |
| Clothing shop support agent | 42 | 6 | 14% |
Many requirements are about what the agent's systems refuse (the claims system won't register a claim on a lapsed policy), and are tested in the agent's own scenarios, not after the fact. Others need checks we haven't built (a date promised in a reply). Each untested requirement is listed by name on the agent's page, so the gap is known and owned.
- Should every rule be required to name at least one requirement, so no rule exists without a reason?
Decided: Yes, as a warning rather than a refusal until a team has a spec. FNOL and the clothing agent already comply. - Which untested FNOL requirements matter most to check after the fact?
Decided: P-PAYOUT-OWED (pay when asked; a status question issues nothing), P-OWN-WORDS (act only on a named claim), P-OWNERSHIP, and the “never a date” half of P-CLAIM-STATUS.