Eval Harness Tutorial · Catalog
Checks by kind of agent
These depend on how the system is built, not on the business. A retrieval-based answerer must cite real sources; a tool-using agent must stop when it's going round in circles. An obligation can belong to several kinds, so it may appear more than once. FNOL and the clothing agent are both A4 (conversational) and A6 (tool-using).
A1 · Single-Turn Transform
One call in, one answer out. No tools, no retrieval, no memory. The model transforms text it was given.
| Id | Level | Must be true | Checked here by |
|---|---|---|---|
| AAC-0017 | MUST | Per-slice accuracy, not just aggregate | — |
| AAC-0018 | SHOULD | Stability under paraphrase and reordering | — |
| AAC-0019 | MUST | Context-window overflow behaviour | — |
| AAC-0020 | SHOULD | Over-refusal rate on in-scope inputs | — |
| AAC-0021 | SHOULD | Position and length bias in the output | — |
A2 · Structured Extractor
Output is a typed object consumed by downstream code rather than read by a human. Separate from A1 because a schema turns quality failures into type failures.
| Id | Level | Must be true | Checked here by |
|---|---|---|---|
| AAC-0022 | MUST | Total schema validity, including nested and optional fields | — |
| AAC-0023 | MUST | Type, unit and format fidelity | — |
| AAC-0024 | MUST | Absent fields are null, never invented | — |
| AAC-0025 | MUST | Field-level precision and recall against gold | — |
| AAC-0026 | SHOULD | Schema evolution does not break consumers | — |
| AAC-0027 | MUST | The downstream consumer is tested on real output | — |
A3 · Grounded Answerer
Retrieval over a corpus, then synthesis. Two systems in a trench coat — the retriever and the generator fail independently and must be tested independently.
| Id | Level | Must be true | Checked here by |
|---|---|---|---|
| AAC-0028 | MUST | Retrieval quality measured on its own | — |
| AAC-0029 | MUST | Every claim is supported by retrieved context | azure_groundedness, bedrock_guardrails, consistency, ragas_faithfulness |
| AAC-0030 | MUST | Citations resolve and actually support the sentence | — |
| AAC-0031 | MUST | Abstention when the corpus cannot answer | — |
| AAC-0032 | MUST | Retrieval respects the caller's permissions | — |
| AAC-0033 | SHOULD | Context precision and noise tolerance | — |
| AAC-0034 | MUST | Index rebuilds are gated by evaluation | — |
| AAC-0035 | SHOULD | Conflicting and stale sources | — |
| AAC-0036 | SHOULD | Injection carried in retrieved documents | — |
| AAC-0108 | MUST | Corpus ingestion is controlled and auditable | — |
| AAC-0112 | MUST | A completed outcome contains the answer | — |
A4 · Conversational Assistant
Multi-turn and stateful. Holds context, memory and user history across turns, but does not take autonomous action.
| Id | Level | Must be true | Checked here by |
|---|---|---|---|
| AAC-0037 | MUST | Goal completion across a full conversation | — |
| AAC-0038 | MUST | State retention and self-consistency | — |
| AAC-0039 | MUST | Correction and topic-switch handling | — |
| AAC-0040 | MUST | Session and tenant isolation | — |
| AAC-0041 | SHOULD | Memory writes are correct and poison-resistant | — |
| AAC-0042 | SHOULD | Conversation-level cost and turns to resolution | — |
| AAC-0043 | MAY | Escalation to a human works under load | decision_vs_should |
| AAC-0112 | MUST | A completed outcome contains the answer | — |
| AAC-0113 | MUST | Action on a stale read | — |
| AAC-0115 | SHOULD | Later outcomes are joined to the run that produced them | — |
| AAC-0118 | MUST | Autonomy widens only on agreement measured for that decision | — |
A5 · Deterministic Workflow
A fixed graph with LLM calls at the nodes. Control flow belongs to your code, so the seams between steps are the new failure surface.
| Id | Level | Must be true | Checked here by |
|---|---|---|---|
| AAC-0044 | MUST | Each node is tested in isolation | — |
| AAC-0045 | MUST | Seams are contract-tested in both directions | — |
| AAC-0046 | MUST | Partial failure and compensating actions | — |
| AAC-0047 | MUST | Retried steps are idempotent | — |
| AAC-0048 | MUST | The graph is deterministic given fixed node outputs | — |
| AAC-0049 | SHOULD | Every branch is exercised | — |
| AAC-0050 | SHOULD | End-to-end budget as the sum of its parts | — |
A6 · Tool-Using Agent
The model chooses which tools to call, in what order, and when to stop. Control flow moves from your code into the model — the single largest jump in test surface in this taxonomy.
| Id | Level | Must be true | Checked here by |
|---|---|---|---|
| AAC-0029 | MUST | Every claim is supported by retrieved context | azure_groundedness, bedrock_guardrails, consistency, ragas_faithfulness |
| AAC-0046 | MUST | Partial failure and compensating actions | — |
| AAC-0047 | MUST | Retried steps are idempotent | — |
| AAC-0051 | MUST | Tool selection accuracy, including no-tool | deepeval_tool_correctness |
| AAC-0052 | MUST | Arguments are valid syntactically and semantically | — |
| AAC-0053 | MUST | Recovery from tool errors and empty results | — |
| AAC-0054 | MUST | Path efficiency against known-good trajectories | — |
| AAC-0055 | MUST | Hard termination under every condition | — |
| AAC-0056 | MUST | Destructive actions are gated | — |
| AAC-0057 | MUST | Least privilege is enforced server-side | — |
| AAC-0058 | MUST | Injection arriving through tool output | — |
| AAC-0059 | SHOULD | Reproducible trajectories with mocked tools | — |
| AAC-0060 | SHOULD | The full trajectory is reconstructable from the trace | — |
| AAC-0076 | MUST | Triggers fire once and only once | — |
| AAC-0105 | SHOULD | Tool results are bounded before they enter context | — |
| AAC-0109 | MUST | Non-productive repetition is detected and broken | — |
| AAC-0110 | MUST | A claimed action is supported by that action's result | claim_needs_action |
| AAC-0112 | MUST | A completed outcome contains the answer | — |
| AAC-0113 | MUST | Action on a stale read | — |
| AAC-0115 | SHOULD | Later outcomes are joined to the run that produced them | — |
| AAC-0118 | MUST | Autonomy widens only on agreement measured for that decision | — |
A7 · Multi-Agent System
Several agents delegating to each other under a supervisor or by handoff. Adds boundary, budget and containment failures on top of A6.
| Id | Level | Must be true | Checked here by |
|---|---|---|---|
| AAC-0061 | MUST | Delegation routes to the right agent | — |
| AAC-0062 | MUST | Handoff payloads are complete | — |
| AAC-0063 | MUST | No loops or ping-pong between agents | — |
| AAC-0064 | MUST | A budget ceiling spans the whole system | — |
| AAC-0065 | MUST | One agent's failure is contained | — |
| AAC-0066 | SHOULD | Contradictory sub-results are reconciled | — |
| AAC-0067 | SHOULD | Per-agent attribution within one trace tree | — |
| AAC-0105 | SHOULD | Tool results are bounded before they enter context | — |
| AAC-0110 | MUST | A claimed action is supported by that action's result | claim_needs_action |
| AAC-0112 | MUST | A completed outcome contains the answer | — |
| AAC-0113 | MUST | Action on a stale read | — |
| AAC-0115 | SHOULD | Later outcomes are joined to the run that produced them | — |
| AAC-0118 | MUST | Autonomy widens only on agreement measured for that decision | — |
A8 · Artifact Producer
Output is code, SQL, config or a document that will be executed or shipped. The artifact can be verified by running it — the strongest signal available anywhere in this catalog.
| Id | Level | Must be true | Checked here by |
|---|---|---|---|
| AAC-0068 | MUST | The artifact parses or compiles | — |
| AAC-0069 | MUST | It executes in a sandbox without error | — |
| AAC-0070 | MUST | It passes a functional test suite | — |
| AAC-0071 | MUST | The generated artifact is security-linted | — |
| AAC-0072 | MUST | Execution is sandboxed and blast radius bounded | — |
| AAC-0073 | SHOULD | Runtime cost of the generated artifact | — |
| AAC-0074 | SHOULD | House style and policy conformance | — |
| AAC-0075 | MUST | Edits stay inside their declared scope | — |
A9 · Autonomous Operator
Runs unattended on a trigger or schedule, with real side effects and nobody watching in real time.
| Id | Level | Must be true | Checked here by |
|---|---|---|---|
| AAC-0076 | MUST | Triggers fire once and only once | — |
| AAC-0077 | MUST | An enforced kill switch on time, tokens and money | — |
| AAC-0078 | MUST | Irreversible actions require approval | decision_vs_should |
| AAC-0079 | MUST | Failure is loud, and so is silence | — |
| AAC-0080 | MUST | Every action is reconstructable after the fact | — |
| AAC-0081 | MUST | Actions can be rolled back | — |
| AAC-0082 | SHOULD | Behaviour on unchanged inputs is watched over time | — |
| AAC-0083 | SHOULD | Blast radius rehearsed in staging | — |
| AAC-0109 | MUST | Non-productive repetition is detected and broken | — |
| AAC-0110 | MUST | A claimed action is supported by that action's result | claim_needs_action |
| AAC-0112 | MUST | A completed outcome contains the answer | — |
| AAC-0113 | MUST | Action on a stale read | — |
| AAC-0115 | SHOULD | Later outcomes are joined to the run that produced them | — |
| AAC-0118 | MUST | Autonomy widens only on agreement measured for that decision | — |
A10 · Classifier · Router · Judge
The model's output steers other machinery: a routing decision, a guardrail verdict, a quality score. Its errors are invisible unless you meta-evaluate them.
| Id | Level | Must be true | Checked here by |
|---|---|---|---|
| AAC-0084 | MUST | Agreement with human labels | — |
| AAC-0085 | MUST | Known judge biases are probed | — |
| AAC-0086 | MUST | Verdict stability across runs and rewording | — |
| AAC-0087 | SHOULD | Confidence scores are calibrated | — |
| AAC-0088 | MUST | Guardrail false-positive and false-negative rates | — |
| AAC-0089 | SHOULD | Routing saves money without losing quality | — |
| AAC-0090 | SHOULD | The judge is re-validated when it changes | — |