cleandataengineering.aiEval Harness, drawn simply
You are here: Architecture › The whole chain

Eval Harness Tutorial · Architecture

The whole chain: from agent code to a dashboard

An agent does its work, writes down what it did, and that record travels to a tool where people look at it. The eval harness sits along that road. This page draws the whole road once, top to bottom, so you can see where each part of the framework stands and which parts are tools you already have.

Read the colours as a status report. Blue is a tool that already exists and isn't ours. Green is ours and built. A dashed white box is ours and not built yet.

1 · THE AGENT HARNESS, AT ITS SMALLESTCustomerasksModel calldecides the next stepTool callorders, claims, CRMReplyto the customerrepeat until donefive moments wherea check can step inbefore modelafter modelbefore toolafter toolbefore replyOpen: Inline checksInline checksallow · change · blockhooks ›every step is recordedTelemetryOpenTelemetry SDK:a span for every modelcall, tool call and turnleaves the agent as2 · WHAT LEAVES THE AGENT: THE SAME ON EVERY PLATFORMOpenTelemetry GenAI spansmodel calls · tool calls · conversation idDecision eventswhich rule allowed, changed or blocked whatOpen: Verdict eventsVerdict eventspass · fail · could not judge; no customer wordsthe format ›sent to3 · WHERE IT LANDS: YOUR TOOLS, DIFFERENT ON EACH PLATFORMApp InsightsAzureCloudWatchAWSPhoenixopen sourceTrace fileOTLP, any collectorLangfuseopen sourceLangSmithLangChainMLflow tracesDatabricksPrometheusoptional: totals onlyEvery store speaks its own query language, so this row can't be made generic.THE EVAL WORKER (OURS)Read tracesrebuild conversationsRules fileYAML, one per agentChecks5 patterns, 5 providersCost rules6 of 36 builtWrite verdictsevents and countersSchedulesampling, backfillreadswritesanswers come from4 · THE COMMON QUESTIONS: ONE LIST FOR EVERY AGENT62 ops questionsthe list is written;12 are answered todayMetric definitionswhat to count, how,split by what: 9 so farone small query writer per store languageKQLApp Insightsrun on a live workspaceLogs InsightsCloudWatchPromQLPrometheusrun in its checkerSQLDuckDB todayshown on5 · WHAT PEOPLE SEE: THEIR SCREENS, OUR CONTENTWeekly reportone page, from verdictsGrafana boardone portable templateAzure Workbookonly if a client asksTheir own screensLangSmith · LangfusePhoenix: scores show upAlertsTeams · ServiceNowAsk in chatover their metric layeran existing tool, not oursours: built and runours: built, not yet run on a live accountours: not built yetA dot on a blue box shows the state of our connector to that tool.
Rows 1, 2 and 4 are the same on every platform. Row 3 is the client's own tool, so it changes with the platform. Row 5 is their screens, filled with content we ship.

The five rows, in plain words

RowWhat happens thereSame on every platform?Our part
1 · Agent harnessThe agent loops: ask the model, call a tool, repeat, reply. A check can step in at five moments. Every step is written down as a span.YesInline checks: built. The loop and the telemetry are the agent's own.
2 · What leaves the agentSpans in the OpenTelemetry GenAI format, plus two kinds of event from us: what a rule decided, and what a check found.Yes. This is the first fixed contract.Verdict events and decision events: built.
3 · Where it landsThe client's trace store. Beside it runs the eval worker: it reads the traces, rebuilds each conversation, runs the rules and writes verdicts back into the same store.No. Each store has its own query language.One reader per store. App Insights is run live; CloudWatch and Phoenix are built and wait for a live run; the rest are not built.
4 · The common questionsOne list of what ops asks of any agent. Each question becomes a definition, and a small query writer turns it into the store's language.The list and the definitions, yes. The query writer, no.62 questions: 12 answered, 7 partly, 43 not yet. 9 metric definitions are written, with query writers for KQL, PromQL and SQL.
5 · What people seeThe client's own screens. We ship what goes on them: a report, a dashboard template, alert rules.The content, yes. The screen is theirs.Weekly report: built. Dashboard templates and alerts: not built.

Prometheus sits in row 3 as an optional second store for totals. Some platforms keep detail and totals in one store, others in two: see one store or two.

Why two contracts make it reusable

The middle of the chain can't be generic, because Azure, AWS and Databricks each store traces their own way. So we fix the two ends instead. Row 2 fixes what goes in. Row 4 fixes what is asked. Between them, a platform needs only a reader, a writer and a query writer.

What every agent gets, and what each agent still writes

The framework does the parts that are the same for every agent. A team adds only what the framework can't know: the agent's own tools, limits and rules.

Comes with the frameworkState
The trace and verdict formats, and rebuilding conversations from tracesBuilt
General checks: personal data, groundedness, tool use, as a ready-made presetBuilt
The logic of the five rule patterns (the team only fills in names and numbers)Built
Rule modes (watch, alert, block), locked rules, the audit listBuilt
Cost rules6 of 36 built
Ops questions that need span metadata only: 32 of 62, with no work per agent once their queries existList written; queries not built
Weekly reportBuilt
Dashboard templates and alert rulesNot built
Each agent's team still doesHow much
Turn on tracing, with a conversation id on every spanOnce, a few lines; no work if the framework already traces
Write its rules file: its tool names, its limits, when to hand over to a personThe claims agent's file is 62 lines and holds 5 rules of its own (read it)
Name its own judge modelOne setting
Say how a conversation ended (resolved, handed over, abandoned)One attribute on the trace
Name an owner per rule, and label about 50 conversations before a rule may blockA few hours per blocking rule
Your review · The whole chain
  1. Is this the drawing you would show a CTO, or should the client version hide the status colours and show only the five rows?
  2. Row 3 lists eight stores. Which three matter most for the clients you expect first? My suggestion: App Insights, Databricks MLflow traces, then CloudWatch.

Reply in chat with the page name and the number, for example “session 2: …”.

Counts are read from src/evalharness/ops_catalog.py and examples/claims-fnol.yaml when this page is built.