Eval Harness Tutorial · Start here
Checking what an AI agent did
An AI agent can write a perfect, polite reply and still be wrong: it can say “your refund has been issued” when no refund was issued, or pay a claim nobody looked up. Reading every conversation by hand doesn't scale. This framework reads what the agent did and checks it against the rules you wrote down.
How to use this tutorial
Read the tabs left to right. Each section teaches first and then shows the code. Pages that need your opinion end in a yellow “Your review” box. The Your review tab collects every question in one place.
Where things stand
| Part | What it does | State |
|---|---|---|
| Sessions | One conversation as an ordered list of steps | Built |
| Five patterns | Reusable rules about a whole conversation, filled in with YAML | Built run on FNOL (29 scenarios) and the clothing agent (49) |
| Provider checks | Presidio, Azure groundedness, DeepEval, Ragas behind one shape | Built |
| Reading traces | Rebuild conversations from standard OpenTelemetry traces | Built from a trace file; Phoenix next |
| Blocking inline | Stop a bad reply before the customer sees it | Not built yet |
| Cost rules | The 36 LLM cost rules (LLM-01 to LLM-36) | Not built yet 6 exist in llm-cost-explainer |
| Writing results back | Scores into Langfuse, App Insights, Prometheus | Not built yet |
The full plan, with every tier and decision, is docs/PLAN.html in the repo.