cleandataengineering.aiEval Harness, drawn simply
You are here: Catalog › By kind of agent

Eval Harness Tutorial · Catalog

Checks by kind of agent

These depend on how the system is built, not on the business. A retrieval-based answerer must cite real sources; a tool-using agent must stop when it's going round in circles. An obligation can belong to several kinds, so it may appear more than once. FNOL and the clothing agent are both A4 (conversational) and A6 (tool-using).

A1 · Single-Turn Transform

One call in, one answer out. No tools, no retrieval, no memory. The model transforms text it was given.

IdLevelMust be trueChecked here by
AAC-0017MUSTPer-slice accuracy, not just aggregate—
AAC-0018SHOULDStability under paraphrase and reordering—
AAC-0019MUSTContext-window overflow behaviour—
AAC-0020SHOULDOver-refusal rate on in-scope inputs—
AAC-0021SHOULDPosition and length bias in the output—

A2 · Structured Extractor

Output is a typed object consumed by downstream code rather than read by a human. Separate from A1 because a schema turns quality failures into type failures.

IdLevelMust be trueChecked here by
AAC-0022MUSTTotal schema validity, including nested and optional fields—
AAC-0023MUSTType, unit and format fidelity—
AAC-0024MUSTAbsent fields are null, never invented—
AAC-0025MUSTField-level precision and recall against gold—
AAC-0026SHOULDSchema evolution does not break consumers—
AAC-0027MUSTThe downstream consumer is tested on real output—

A3 · Grounded Answerer

Retrieval over a corpus, then synthesis. Two systems in a trench coat — the retriever and the generator fail independently and must be tested independently.

IdLevelMust be trueChecked here by
AAC-0028MUSTRetrieval quality measured on its own—
AAC-0029MUSTEvery claim is supported by retrieved contextazure_groundedness, bedrock_guardrails, consistency, ragas_faithfulness
AAC-0030MUSTCitations resolve and actually support the sentence—
AAC-0031MUSTAbstention when the corpus cannot answer—
AAC-0032MUSTRetrieval respects the caller's permissions—
AAC-0033SHOULDContext precision and noise tolerance—
AAC-0034MUSTIndex rebuilds are gated by evaluation—
AAC-0035SHOULDConflicting and stale sources—
AAC-0036SHOULDInjection carried in retrieved documents—
AAC-0108MUSTCorpus ingestion is controlled and auditable—
AAC-0112MUSTA completed outcome contains the answer—

A4 · Conversational Assistant

Multi-turn and stateful. Holds context, memory and user history across turns, but does not take autonomous action.

IdLevelMust be trueChecked here by
AAC-0037MUSTGoal completion across a full conversation—
AAC-0038MUSTState retention and self-consistency—
AAC-0039MUSTCorrection and topic-switch handling—
AAC-0040MUSTSession and tenant isolation—
AAC-0041SHOULDMemory writes are correct and poison-resistant—
AAC-0042SHOULDConversation-level cost and turns to resolution—
AAC-0043MAYEscalation to a human works under loaddecision_vs_should
AAC-0112MUSTA completed outcome contains the answer—
AAC-0113MUSTAction on a stale read—
AAC-0115SHOULDLater outcomes are joined to the run that produced them—
AAC-0118MUSTAutonomy widens only on agreement measured for that decision—

A5 · Deterministic Workflow

A fixed graph with LLM calls at the nodes. Control flow belongs to your code, so the seams between steps are the new failure surface.

IdLevelMust be trueChecked here by
AAC-0044MUSTEach node is tested in isolation—
AAC-0045MUSTSeams are contract-tested in both directions—
AAC-0046MUSTPartial failure and compensating actions—
AAC-0047MUSTRetried steps are idempotent—
AAC-0048MUSTThe graph is deterministic given fixed node outputs—
AAC-0049SHOULDEvery branch is exercised—
AAC-0050SHOULDEnd-to-end budget as the sum of its parts—

A6 · Tool-Using Agent

The model chooses which tools to call, in what order, and when to stop. Control flow moves from your code into the model — the single largest jump in test surface in this taxonomy.

IdLevelMust be trueChecked here by
AAC-0029MUSTEvery claim is supported by retrieved contextazure_groundedness, bedrock_guardrails, consistency, ragas_faithfulness
AAC-0046MUSTPartial failure and compensating actions—
AAC-0047MUSTRetried steps are idempotent—
AAC-0051MUSTTool selection accuracy, including no-tooldeepeval_tool_correctness
AAC-0052MUSTArguments are valid syntactically and semantically—
AAC-0053MUSTRecovery from tool errors and empty results—
AAC-0054MUSTPath efficiency against known-good trajectories—
AAC-0055MUSTHard termination under every condition—
AAC-0056MUSTDestructive actions are gated—
AAC-0057MUSTLeast privilege is enforced server-side—
AAC-0058MUSTInjection arriving through tool output—
AAC-0059SHOULDReproducible trajectories with mocked tools—
AAC-0060SHOULDThe full trajectory is reconstructable from the trace—
AAC-0076MUSTTriggers fire once and only once—
AAC-0105SHOULDTool results are bounded before they enter context—
AAC-0109MUSTNon-productive repetition is detected and broken—
AAC-0110MUSTA claimed action is supported by that action's resultclaim_needs_action
AAC-0112MUSTA completed outcome contains the answer—
AAC-0113MUSTAction on a stale read—
AAC-0115SHOULDLater outcomes are joined to the run that produced them—
AAC-0118MUSTAutonomy widens only on agreement measured for that decision—

A7 · Multi-Agent System

Several agents delegating to each other under a supervisor or by handoff. Adds boundary, budget and containment failures on top of A6.

IdLevelMust be trueChecked here by
AAC-0061MUSTDelegation routes to the right agent—
AAC-0062MUSTHandoff payloads are complete—
AAC-0063MUSTNo loops or ping-pong between agents—
AAC-0064MUSTA budget ceiling spans the whole system—
AAC-0065MUSTOne agent's failure is contained—
AAC-0066SHOULDContradictory sub-results are reconciled—
AAC-0067SHOULDPer-agent attribution within one trace tree—
AAC-0105SHOULDTool results are bounded before they enter context—
AAC-0110MUSTA claimed action is supported by that action's resultclaim_needs_action
AAC-0112MUSTA completed outcome contains the answer—
AAC-0113MUSTAction on a stale read—
AAC-0115SHOULDLater outcomes are joined to the run that produced them—
AAC-0118MUSTAutonomy widens only on agreement measured for that decision—

A8 · Artifact Producer

Output is code, SQL, config or a document that will be executed or shipped. The artifact can be verified by running it — the strongest signal available anywhere in this catalog.

IdLevelMust be trueChecked here by
AAC-0068MUSTThe artifact parses or compiles—
AAC-0069MUSTIt executes in a sandbox without error—
AAC-0070MUSTIt passes a functional test suite—
AAC-0071MUSTThe generated artifact is security-linted—
AAC-0072MUSTExecution is sandboxed and blast radius bounded—
AAC-0073SHOULDRuntime cost of the generated artifact—
AAC-0074SHOULDHouse style and policy conformance—
AAC-0075MUSTEdits stay inside their declared scope—

A9 · Autonomous Operator

Runs unattended on a trigger or schedule, with real side effects and nobody watching in real time.

IdLevelMust be trueChecked here by
AAC-0076MUSTTriggers fire once and only once—
AAC-0077MUSTAn enforced kill switch on time, tokens and money—
AAC-0078MUSTIrreversible actions require approvaldecision_vs_should
AAC-0079MUSTFailure is loud, and so is silence—
AAC-0080MUSTEvery action is reconstructable after the fact—
AAC-0081MUSTActions can be rolled back—
AAC-0082SHOULDBehaviour on unchanged inputs is watched over time—
AAC-0083SHOULDBlast radius rehearsed in staging—
AAC-0109MUSTNon-productive repetition is detected and broken—
AAC-0110MUSTA claimed action is supported by that action's resultclaim_needs_action
AAC-0112MUSTA completed outcome contains the answer—
AAC-0113MUSTAction on a stale read—
AAC-0115SHOULDLater outcomes are joined to the run that produced them—
AAC-0118MUSTAutonomy widens only on agreement measured for that decision—

A10 · Classifier · Router · Judge

The model's output steers other machinery: a routing decision, a guardrail verdict, a quality score. Its errors are invisible unless you meta-evaluate them.

IdLevelMust be trueChecked here by
AAC-0084MUSTAgreement with human labels—
AAC-0085MUSTKnown judge biases are probed—
AAC-0086MUSTVerdict stability across runs and rewording—
AAC-0087SHOULDConfidence scores are calibrated—
AAC-0088MUSTGuardrail false-positive and false-negative rates—
AAC-0089SHOULDRouting saves money without losing quality—
AAC-0090SHOULDThe judge is re-validated when it changes—