Eval Harness Tutorial · Catalog
A catalog of common checks
You asked whether there's a corpus of common checks, sorted into domain-agnostic and domain-specific. There is now: this section. It isn't typed in. It is read from the sources that already hold the checks, when the pages are built:
- The AI Assurance Catalog (our open standard): 118 obligations, things that must be true of an AI system. 36 apply to every AI system; 82 apply to a kind of system.
- The LLM cost rules: 36 ways money is wasted on model calls.
- Warning signs (the catalog's detection patterns): 56 symptoms seen in real traces, each linked to the obligations that catch it.
- Our five patterns, filled in with each business's rules from its spec.
How much of it we check today
| Layer | In the catalog | Checked by this framework today |
|---|---|---|
| Any agent: obligations | 36 | 1 |
| Any agent: cost rules | 36 | 6 (built in llm-cost-explainer; joining here in Tier 5) |
| By kind of agent | 82 | 5 |
| For one business (FNOL, clothing) | 84 requirements in their specs | 14 tested by 12 rules |
Read the gaps honestly
Most obligations aren't covered by a check here yet, and that's expected: many are tested before release (golden sets, red-teaming), some belong to the platform (identity, network), and the framework is young. Every “—” on these pages is a known, named gap, not a hidden one.
Decided · The catalog
- Is “any agent / by kind of agent / for one business” the right way to sort checks for a client conversation?
Decided: Yes, keep the three layers. - Which gaps should we close first: the core obligations, the cost rules, or more domain requirements?
Decided: Domain requirements first (that's what a client pays for), then the cost rules (Tier 5), then the core obligations that can be checked after the fact.
Look inside
Checks for any agentCore obligations, cost rules and warning signs: domain-agnostic.Checks by kind of agentObligations that depend on how the system is built: RAG, tools, multi-agent…Checks for one businessThe five pattern shapes, filled in with each business's own rules.Ops questions answered from traces62 questions an ops person asks of agent traces, in 12 groups: the spec for saved questions and dashboard packs.