cleandataengineering.aiEval Harness, drawn simply
You are here: Catalog › Any agent

Eval Harness Tutorial · Catalog

Checks for any agent

These apply whatever the business: a support bot, a claims agent, a code assistant. MUST means required, SHOULD recommended, MAY optional.

36 core obligations, by concern

Correctness

IdLevelMust be trueChecked here by
AAC-0001MUSTTask accuracy on a frozen golden set—
AAC-0014SHOULDContinuous scoring of sampled production traffic—
AAC-0098MUSTEvery reachable model is evaluated, not just the primary—

Cost

IdLevelMust be trueChecked here by
AAC-0008MUSTCost per successful task, not per call—
AAC-0093SHOULDPer-caller quota and spend ceilings are enforced—
AAC-0102MUSTCost regression is gated at release—
AAC-0103SHOULDContext growth is tracked across releases—
AAC-0104SHOULDSpend is attributable to tenant, feature and route—

Determinism

IdLevelMust be trueChecked here by
AAC-0010SHOULDVariance across repeated runs is bounded—

Drift

IdLevelMust be trueChecked here by
AAC-0012MUSTModel and prompt versions are pinned and rollback is exercised—
AAC-0013MUSTRegression against the previous release—
AAC-0016SHOULDScheduled re-run against unchanged inputs—
AAC-0101MUSTRouting changes are gated like model changes—

Format

IdLevelMust be trueChecked here by
AAC-0002MUSTOutput conforms to its declared contract—
AAC-0099MUSTThe output contract holds on every route—

Instruction

IdLevelMust be trueChecked here by
AAC-0003MUSTInstruction and constraint adherence—

Latency

IdLevelMust be trueChecked here by
AAC-0007MUSTLatency within budget at expected concurrency—

Observability

IdLevelMust be trueChecked here by
AAC-0011MUSTEvery call emits a complete trace—
AAC-0100MUSTThe serving route is recorded, with its reason and its cost—
AAC-0114SHOULDBehaviour rates are watched in production against a baseline—

Oversight

IdLevelMust be trueChecked here by
AAC-0094SHOULDOnly approved models are reachable—

Privacy

IdLevelMust be trueChecked here by
AAC-0006MUSTNo leakage of PII, secrets or credentialsbedrock_guardrails, presidio_pii
AAC-0095MUSTLogged prompts and responses are redacted and retention-bounded—
AAC-0096MAYCached responses never cross a trust boundary—
AAC-0097MAYProcessing region is enforced and recorded—
AAC-0117MUSTWhat is held about one person can be found and removed—

Reliability

IdLevelMust be trueChecked here by
AAC-0009MUSTGraceful degradation on provider failure—
AAC-0116SHOULDA synthetic run exercises the deployed system on a schedule—

Robustness

IdLevelMust be trueChecked here by
AAC-0015MUSTDefined behaviour on degenerate input—

Safety

IdLevelMust be trueChecked here by
AAC-0005MUSTSafety and policy compliance under adversarial input—
AAC-0091MUSTGuardrail enforcement fails closed—
AAC-0092MAYStreamed output is screened before it reaches the caller—

Security

IdLevelMust be trueChecked here by
AAC-0004MUSTResistance to injection from untrusted content—
AAC-0106MUSTThe system prompt is not a security boundary—
AAC-0107MUSTServing artifacts are the ones that were evaluated—
AAC-0111MUSTA request acts only as the identity its credential proves—

36 cost rules

IdThe waste it findsState
LLM-01Long repeated prompt start, not cachedBuilt in llm-cost-explainer
LLM-02Cache broken by a changing prefixNot built yet
LLM-03Identical request paid for twiceNot built yet
LLM-04Same text embedded againNot built yet
LLM-05Big model for a small jobNot built yet
LLM-06Bulk job on the real-time APIBuilt in llm-cost-explainer
LLM-07Too much thinking for simple tasksNot built yet
LLM-08Silent switch to a pricier modelBuilt in llm-cost-explainer
LLM-09Conversation history never trimmedNot built yet
LLM-10Over-retrieval in RAGNot built yet
LLM-11Every tool definition sent on every callNot built yet
LLM-12Large fixed examples in every promptNot built yet
LLM-13Images sent at full resolutionNot built yet
LLM-14Retry stormNot built yet
LLM-15Agent stuck in a loopNot built yet
LLM-16No output limitNot built yet
LLM-17Paying for answers nobody receivesNot built yet
LLM-18Customer costs more than they payNot built yet
LLM-19One user or bot drives most of the costNot built yet
LLM-20A cost jump nobody explainedNot built yet
LLM-21Provisioned or committed capacity sitting idle , or steady high volume with no commitmentNot built yet
LLM-22Priority tier used for work that isn't urgentBuilt in llm-cost-explainer
LLM-23Spend nobody can attributeNot built yet
LLM-24Old model still in use where a cheaper successor existsBuilt in llm-cost-explainer
LLM-25Several answers generated, one usedNot built yet
LLM-26Cache lifetime too short for the trafficBuilt in llm-cost-explainer
LLM-27Retries caused by unparseable outputNot built yet
LLM-28Sub-agents each re-sent the full contextNot built yet
LLM-29LLM doing work ordinary code could doNot built yet
LLM-30A chain of calls that could be oneNot built yet
LLM-31Guardrail / judge taxNot built yet
LLM-32Polling unchanged inputNot built yet
LLM-33Calls on every keystroke / regenerate burstsNot built yet
LLM-34Background AI for users who never lookNot built yet
LLM-35Dev, test, CI or load-test traffic on production keysNot built yet
LLM-36Cost regression shipped in a releaseNot built yet

56 warning signs

Symptoms you can spot in traces, each linked to the obligations that catch it. They're grouped by where they show up: the agent's behaviour, cost, or operations.

IdAreaWarning signCaught by
AACP-0013agentPrompt-only guardrailsAAC-0055, AAC-0056, AAC-0057
AACP-0014agentTool-error ping-pongAAC-0053, AAC-0055, AAC-0054
AACP-0015agentInstructions arriving through tool outputAAC-0058, AAC-0036, AAC-0004
AACP-0016agentContext lost at handoffAAC-0062, AAC-0061
AACP-0017agentSupervisor ping-pongAAC-0063, AAC-0064
AACP-0018agentTrajectory judged only by its answerAAC-0054, AAC-0060, AAC-0100
AACP-0001costRetry amplificationAAC-0008, AAC-0102, AAC-0104
AACP-0002costContext accretionAAC-0103
AACP-0003costConversation tail costAAC-0042, AAC-0103
AACP-0004costTool-output dumpingAAC-0105, AAC-0054
AACP-0005costFrontier-model defaultAAC-0089, AAC-0102
AACP-0006costCancellation is not propagated—
AACP-0007costVolatile prompt prefix defeats caching—
AACP-0008costUnbounded fan-out under a supervisorAAC-0064, AAC-0067
AACP-0009costUngoverned direct provider accessAAC-0094, AAC-0104
AACP-0010costJudge cost exceeds the system it judgesAAC-0014, AAC-0104
AACP-0011costUnevaluated cheap routeAAC-0098, AAC-0099, AAC-0100
AACP-0012costCost per call, not per taskAAC-0008, AAC-0050
AACP-0019operationsRetry after the far end said noAAC-0053, AAC-0047
AACP-0020operationsRetry storm on a failing dependencyAAC-0053, AAC-0009
AACP-0021operationsWrite without a readAAC-0113, AAC-0056
AACP-0022operationsInvented argumentAAC-0052
AACP-0023operationsA tool that is not offeredAAC-0051
AACP-0024operationsArguments that fail their schema, againAAC-0052
AACP-0025operationsAn answer built on a truncated resultAAC-0105
AACP-0026operationsA dead tool on the surfaceAAC-0103, AAC-0051
AACP-0027operationsTool latency driftAAC-0007
AACP-0028operationsReply contradicts a tool resultAAC-0110, AAC-0113
AACP-0029operationsA claimed action without its effectAAC-0110
AACP-0030operationsAn identifier nobody gaveAAC-0110, AAC-0024
AACP-0031operationsInternal text in a replyAAC-0106, AAC-0002
AACP-0032operationsAsks for what it was already givenAAC-0038
AACP-0033operationsPersonal data in a replyAAC-0006
AACP-0034operationsA completion with nothing in itAAC-0112
AACP-0035operationsApology without progressAAC-0037
AACP-0036operationsEnded by a limit, not by an answerAAC-0055, AAC-0077
AACP-0037operationsA longer path than the task needsAAC-0054
AACP-0038operationsUnreadable model outputAAC-0002
AACP-0039operationsAnswered over a failed toolAAC-0053
AACP-0040operationsThe user repeats themselvesAAC-0039, AAC-0037
AACP-0041operationsBack within a dayAAC-0115, AAC-0042
AACP-0042operationsA person asked for right after an answerAAC-0115
AACP-0043operationsTurns to resolution creeping upAAC-0042
AACP-0044operationsEscalations nobody neededAAC-0020, AAC-0043
AACP-0045operationsWaits that lapse with nobodyAAC-0043, AAC-0078
AACP-0046operationsOne rule refusing too muchAAC-0020, AAC-0088
AACP-0047operationsAuthority reached forAAC-0057, AAC-0111
AACP-0048operationsA promise nothing is keepingAAC-0112
AACP-0049operationsHealthy status codes, broken behaviourAAC-0114
AACP-0050operationsA quiet deploymentAAC-0116, AAC-0079
AACP-0051operationsLatency tail over budgetAAC-0007
AACP-0052operationsProvider degradationAAC-0009
AACP-0053operationsSpend outrunning its budgetAAC-0008, AAC-0093, AAC-0077
AACP-0054operationsThe served model is not the pinned oneAAC-0012, AAC-0094
AACP-0055operationsA release that moved the ratesAAC-0013, AAC-0101
AACP-0056operationsTelemetry that silently dropsAAC-0011, AAC-0079