cleandataengineering.aiEval Harness, drawn simply
You are here: Start here › The whole picture

Eval Harness Tutorial · Start here

The whole picture, one step at a time

ReadGet what the agentdid: from a tracefile, Phoenix,Langfuse or AppInsights.1RebuildTurn the raw recordsinto one Session perconversation.2PlanRead your YAML:which checks, withwhich settings.3CheckRun each check oneach session. Eachcheck answers in thesame result shape.4Decide and recordTurn results into anaction (block,alert), write themback, send a weeklyreport.5Solid boxes are built. The dashed one is not built yet.
The framework is these five steps. Each step talks to the next through a fixed shape, so any piece can be swapped.

Why the steps are separate. Each step meets the next through a fixed shape (a port). A Phoenix reader and an App Insights reader both produce the same Session, so the checks never know where the data came from. A Presidio check and one of our patterns both answer with the same result, so the reports never know who judged. Swapping a tool means adding one adapter and changing one line of YAML.

Why “decide” is its own step. A check only says what it found. Whether that blocks a reply, raises an alert or is just counted is a separate choice, so the same check can block in one company and only alert in another.

We never replace a team's tools. No database, dashboard or chat UI of our own: we read from where traces already are and write verdicts back there.

Decided · The whole picture
  1. Is the five-step picture clear enough to explain to an ops person in two minutes? If not, which step is unclear?
    Decided: Yes. The five steps stay the explanation.
  2. Is anything missing that an ops team would expect, such as a weekly report or alerts?
    Decided: Yes: a weekly report and alerts. Both belong to step 5 (decide and record), now drawn there as not built yet.