Eval Harness Tutorial · Catalog
Checks for any agent
These apply whatever the business: a support bot, a claims agent, a code assistant. MUST means required, SHOULD recommended, MAY optional.
36 core obligations, by concern
Correctness
| Id | Level | Must be true | Checked here by |
|---|---|---|---|
| AAC-0001 | MUST | Task accuracy on a frozen golden set | — |
| AAC-0014 | SHOULD | Continuous scoring of sampled production traffic | — |
| AAC-0098 | MUST | Every reachable model is evaluated, not just the primary | — |
Cost
| Id | Level | Must be true | Checked here by |
|---|---|---|---|
| AAC-0008 | MUST | Cost per successful task, not per call | — |
| AAC-0093 | SHOULD | Per-caller quota and spend ceilings are enforced | — |
| AAC-0102 | MUST | Cost regression is gated at release | — |
| AAC-0103 | SHOULD | Context growth is tracked across releases | — |
| AAC-0104 | SHOULD | Spend is attributable to tenant, feature and route | — |
Determinism
| Id | Level | Must be true | Checked here by |
|---|---|---|---|
| AAC-0010 | SHOULD | Variance across repeated runs is bounded | — |
Drift
| Id | Level | Must be true | Checked here by |
|---|---|---|---|
| AAC-0012 | MUST | Model and prompt versions are pinned and rollback is exercised | — |
| AAC-0013 | MUST | Regression against the previous release | — |
| AAC-0016 | SHOULD | Scheduled re-run against unchanged inputs | — |
| AAC-0101 | MUST | Routing changes are gated like model changes | — |
Format
| Id | Level | Must be true | Checked here by |
|---|---|---|---|
| AAC-0002 | MUST | Output conforms to its declared contract | — |
| AAC-0099 | MUST | The output contract holds on every route | — |
Instruction
| Id | Level | Must be true | Checked here by |
|---|---|---|---|
| AAC-0003 | MUST | Instruction and constraint adherence | — |
Latency
| Id | Level | Must be true | Checked here by |
|---|---|---|---|
| AAC-0007 | MUST | Latency within budget at expected concurrency | — |
Observability
| Id | Level | Must be true | Checked here by |
|---|---|---|---|
| AAC-0011 | MUST | Every call emits a complete trace | — |
| AAC-0100 | MUST | The serving route is recorded, with its reason and its cost | — |
| AAC-0114 | SHOULD | Behaviour rates are watched in production against a baseline | — |
Oversight
| Id | Level | Must be true | Checked here by |
|---|---|---|---|
| AAC-0094 | SHOULD | Only approved models are reachable | — |
Privacy
| Id | Level | Must be true | Checked here by |
|---|---|---|---|
| AAC-0006 | MUST | No leakage of PII, secrets or credentials | bedrock_guardrails, presidio_pii |
| AAC-0095 | MUST | Logged prompts and responses are redacted and retention-bounded | — |
| AAC-0096 | MAY | Cached responses never cross a trust boundary | — |
| AAC-0097 | MAY | Processing region is enforced and recorded | — |
| AAC-0117 | MUST | What is held about one person can be found and removed | — |
Reliability
| Id | Level | Must be true | Checked here by |
|---|---|---|---|
| AAC-0009 | MUST | Graceful degradation on provider failure | — |
| AAC-0116 | SHOULD | A synthetic run exercises the deployed system on a schedule | — |
Robustness
| Id | Level | Must be true | Checked here by |
|---|---|---|---|
| AAC-0015 | MUST | Defined behaviour on degenerate input | — |
Safety
| Id | Level | Must be true | Checked here by |
|---|---|---|---|
| AAC-0005 | MUST | Safety and policy compliance under adversarial input | — |
| AAC-0091 | MUST | Guardrail enforcement fails closed | — |
| AAC-0092 | MAY | Streamed output is screened before it reaches the caller | — |
Security
| Id | Level | Must be true | Checked here by |
|---|---|---|---|
| AAC-0004 | MUST | Resistance to injection from untrusted content | — |
| AAC-0106 | MUST | The system prompt is not a security boundary | — |
| AAC-0107 | MUST | Serving artifacts are the ones that were evaluated | — |
| AAC-0111 | MUST | A request acts only as the identity its credential proves | — |
36 cost rules
| Id | The waste it finds | State |
|---|---|---|
| LLM-01 | Long repeated prompt start, not cached | Built in llm-cost-explainer |
| LLM-02 | Cache broken by a changing prefix | Not built yet |
| LLM-03 | Identical request paid for twice | Not built yet |
| LLM-04 | Same text embedded again | Not built yet |
| LLM-05 | Big model for a small job | Not built yet |
| LLM-06 | Bulk job on the real-time API | Built in llm-cost-explainer |
| LLM-07 | Too much thinking for simple tasks | Not built yet |
| LLM-08 | Silent switch to a pricier model | Built in llm-cost-explainer |
| LLM-09 | Conversation history never trimmed | Not built yet |
| LLM-10 | Over-retrieval in RAG | Not built yet |
| LLM-11 | Every tool definition sent on every call | Not built yet |
| LLM-12 | Large fixed examples in every prompt | Not built yet |
| LLM-13 | Images sent at full resolution | Not built yet |
| LLM-14 | Retry storm | Not built yet |
| LLM-15 | Agent stuck in a loop | Not built yet |
| LLM-16 | No output limit | Not built yet |
| LLM-17 | Paying for answers nobody receives | Not built yet |
| LLM-18 | Customer costs more than they pay | Not built yet |
| LLM-19 | One user or bot drives most of the cost | Not built yet |
| LLM-20 | A cost jump nobody explained | Not built yet |
| LLM-21 | Provisioned or committed capacity sitting idle , or steady high volume with no commitment | Not built yet |
| LLM-22 | Priority tier used for work that isn't urgent | Built in llm-cost-explainer |
| LLM-23 | Spend nobody can attribute | Not built yet |
| LLM-24 | Old model still in use where a cheaper successor exists | Built in llm-cost-explainer |
| LLM-25 | Several answers generated, one used | Not built yet |
| LLM-26 | Cache lifetime too short for the traffic | Built in llm-cost-explainer |
| LLM-27 | Retries caused by unparseable output | Not built yet |
| LLM-28 | Sub-agents each re-sent the full context | Not built yet |
| LLM-29 | LLM doing work ordinary code could do | Not built yet |
| LLM-30 | A chain of calls that could be one | Not built yet |
| LLM-31 | Guardrail / judge tax | Not built yet |
| LLM-32 | Polling unchanged input | Not built yet |
| LLM-33 | Calls on every keystroke / regenerate bursts | Not built yet |
| LLM-34 | Background AI for users who never look | Not built yet |
| LLM-35 | Dev, test, CI or load-test traffic on production keys | Not built yet |
| LLM-36 | Cost regression shipped in a release | Not built yet |
56 warning signs
Symptoms you can spot in traces, each linked to the obligations that catch it. They're grouped by where they show up: the agent's behaviour, cost, or operations.
| Id | Area | Warning sign | Caught by |
|---|---|---|---|
| AACP-0013 | agent | Prompt-only guardrails | AAC-0055, AAC-0056, AAC-0057 |
| AACP-0014 | agent | Tool-error ping-pong | AAC-0053, AAC-0055, AAC-0054 |
| AACP-0015 | agent | Instructions arriving through tool output | AAC-0058, AAC-0036, AAC-0004 |
| AACP-0016 | agent | Context lost at handoff | AAC-0062, AAC-0061 |
| AACP-0017 | agent | Supervisor ping-pong | AAC-0063, AAC-0064 |
| AACP-0018 | agent | Trajectory judged only by its answer | AAC-0054, AAC-0060, AAC-0100 |
| AACP-0001 | cost | Retry amplification | AAC-0008, AAC-0102, AAC-0104 |
| AACP-0002 | cost | Context accretion | AAC-0103 |
| AACP-0003 | cost | Conversation tail cost | AAC-0042, AAC-0103 |
| AACP-0004 | cost | Tool-output dumping | AAC-0105, AAC-0054 |
| AACP-0005 | cost | Frontier-model default | AAC-0089, AAC-0102 |
| AACP-0006 | cost | Cancellation is not propagated | — |
| AACP-0007 | cost | Volatile prompt prefix defeats caching | — |
| AACP-0008 | cost | Unbounded fan-out under a supervisor | AAC-0064, AAC-0067 |
| AACP-0009 | cost | Ungoverned direct provider access | AAC-0094, AAC-0104 |
| AACP-0010 | cost | Judge cost exceeds the system it judges | AAC-0014, AAC-0104 |
| AACP-0011 | cost | Unevaluated cheap route | AAC-0098, AAC-0099, AAC-0100 |
| AACP-0012 | cost | Cost per call, not per task | AAC-0008, AAC-0050 |
| AACP-0019 | operations | Retry after the far end said no | AAC-0053, AAC-0047 |
| AACP-0020 | operations | Retry storm on a failing dependency | AAC-0053, AAC-0009 |
| AACP-0021 | operations | Write without a read | AAC-0113, AAC-0056 |
| AACP-0022 | operations | Invented argument | AAC-0052 |
| AACP-0023 | operations | A tool that is not offered | AAC-0051 |
| AACP-0024 | operations | Arguments that fail their schema, again | AAC-0052 |
| AACP-0025 | operations | An answer built on a truncated result | AAC-0105 |
| AACP-0026 | operations | A dead tool on the surface | AAC-0103, AAC-0051 |
| AACP-0027 | operations | Tool latency drift | AAC-0007 |
| AACP-0028 | operations | Reply contradicts a tool result | AAC-0110, AAC-0113 |
| AACP-0029 | operations | A claimed action without its effect | AAC-0110 |
| AACP-0030 | operations | An identifier nobody gave | AAC-0110, AAC-0024 |
| AACP-0031 | operations | Internal text in a reply | AAC-0106, AAC-0002 |
| AACP-0032 | operations | Asks for what it was already given | AAC-0038 |
| AACP-0033 | operations | Personal data in a reply | AAC-0006 |
| AACP-0034 | operations | A completion with nothing in it | AAC-0112 |
| AACP-0035 | operations | Apology without progress | AAC-0037 |
| AACP-0036 | operations | Ended by a limit, not by an answer | AAC-0055, AAC-0077 |
| AACP-0037 | operations | A longer path than the task needs | AAC-0054 |
| AACP-0038 | operations | Unreadable model output | AAC-0002 |
| AACP-0039 | operations | Answered over a failed tool | AAC-0053 |
| AACP-0040 | operations | The user repeats themselves | AAC-0039, AAC-0037 |
| AACP-0041 | operations | Back within a day | AAC-0115, AAC-0042 |
| AACP-0042 | operations | A person asked for right after an answer | AAC-0115 |
| AACP-0043 | operations | Turns to resolution creeping up | AAC-0042 |
| AACP-0044 | operations | Escalations nobody needed | AAC-0020, AAC-0043 |
| AACP-0045 | operations | Waits that lapse with nobody | AAC-0043, AAC-0078 |
| AACP-0046 | operations | One rule refusing too much | AAC-0020, AAC-0088 |
| AACP-0047 | operations | Authority reached for | AAC-0057, AAC-0111 |
| AACP-0048 | operations | A promise nothing is keeping | AAC-0112 |
| AACP-0049 | operations | Healthy status codes, broken behaviour | AAC-0114 |
| AACP-0050 | operations | A quiet deployment | AAC-0116, AAC-0079 |
| AACP-0051 | operations | Latency tail over budget | AAC-0007 |
| AACP-0052 | operations | Provider degradation | AAC-0009 |
| AACP-0053 | operations | Spend outrunning its budget | AAC-0008, AAC-0093, AAC-0077 |
| AACP-0054 | operations | The served model is not the pinned one | AAC-0012, AAC-0094 |
| AACP-0055 | operations | A release that moved the rates | AAC-0013, AAC-0101 |
| AACP-0056 | operations | Telemetry that silently drops | AAC-0011, AAC-0079 |