cleandataengineering.aiEval Harness, drawn simply
You are here: Start here

Eval Harness Tutorial · Start here

Checking what an AI agent did

An AI agent can write a perfect, polite reply and still be wrong: it can say “your refund has been issued” when no refund was issued, or pay a claim nobody looked up. Reading every conversation by hand doesn't scale. This framework reads what the agent did and checks it against the rules you wrote down.

Your agenttalks to customers,calls toolswritesTraceswhat happened, stepby step (it alreadywrites these)we readOpen: evalharnessevalharnessrebuilds eachconversation, runsthe checks youwrote in YAMLstart here ›Resultspass / fail + why,back into toolsyou already useChecks come from: our five patterns, plus Presidio, Azure, DeepEval, Ragas…
The agent already records what it does (traces). We read those, rebuild each conversation, run your checks, and write the verdicts back where your team already looks.
How to use this tutorial

Read the tabs left to right. Each section teaches first and then shows the code. Pages that need your opinion end in a yellow “Your review” box. The Your review tab collects every question in one place.

Where things stand

PartWhat it doesState
SessionsOne conversation as an ordered list of stepsBuilt
Five patternsReusable rules about a whole conversation, filled in with YAMLBuilt run on FNOL (29 scenarios) and the clothing agent (49)
Provider checksPresidio, Azure groundedness, DeepEval, Ragas behind one shapeBuilt
Reading tracesRebuild conversations from standard OpenTelemetry tracesBuilt from a trace file; Phoenix next
Blocking inlineStop a bad reply before the customer sees itNot built yet
Cost rulesThe 36 LLM cost rules (LLM-01 to LLM-36)Not built yet 6 exist in llm-cost-explainer
Writing results backScores into Langfuse, App Insights, PrometheusNot built yet

The full plan, with every tier and decision, is docs/PLAN.html in the repo.

Look inside