cleandataengineering.aiEval Harness, drawn simply
You are here: Catalog

Eval Harness Tutorial · Catalog

A catalog of common checks

You asked whether there's a corpus of common checks, sorted into domain-agnostic and domain-specific. There is now: this section. It isn't typed in. It is read from the sources that already hold the checks, when the pages are built:

  • The AI Assurance Catalog (our open standard): 118 obligations, things that must be true of an AI system. 36 apply to every AI system; 82 apply to a kind of system.
  • The LLM cost rules: 36 ways money is wasted on model calls.
  • Warning signs (the catalog's detection patterns): 56 symptoms seen in real traces, each linked to the obligations that catch it.
  • Our five patterns, filled in with each business's rules from its spec.
From most general to most specific. Click a layer to see every check in it.

How much of it we check today

LayerIn the catalogChecked by this framework today
Any agent: obligations361
Any agent: cost rules366 (built in llm-cost-explainer; joining here in Tier 5)
By kind of agent825
For one business (FNOL, clothing)84 requirements in their specs14 tested by 12 rules
Read the gaps honestly

Most obligations aren't covered by a check here yet, and that's expected: many are tested before release (golden sets, red-teaming), some belong to the platform (identity, network), and the framework is young. Every “—” on these pages is a known, named gap, not a hidden one.

Decided · The catalog
  1. Is “any agent / by kind of agent / for one business” the right way to sort checks for a client conversation?
    Decided: Yes, keep the three layers.
  2. Which gaps should we close first: the core obligations, the cost rules, or more domain requirements?
    Decided: Domain requirements first (that's what a client pays for), then the cost rules (Tier 5), then the core obligations that can be checked after the fact.

Look inside