cleandataengineering.aiEval Harness, drawn simply
You are here: Architecture

Eval Harness Tutorial · Architecture

Where the framework fits

This is what we're trying to do, in one drawing. Your agent and the tools around it stay as they are. The framework (the dark box) plugs in at four points, numbered below, and writes its verdicts back into tools your team already uses. It has no database, dashboard or chat window of its own.

YOUR AI APPLICATION (unchanged)Customertypes a messageYour agentPydantic AI · LangGraphAgent SDK · own codeGatewayLiteLLM · Azure APIMModelsOpenAI · Azure · GroqTools / systemsorders, claims, CRMInstrumentorsOpenTelemetry SDK,OpenInference,the framework's owntracingOpen: evalharnessevalharnessread · rebuild sessions · plan (YAML)check · decide · recordone request shape, one result shapeinside ›1inline2gatewayWhere traces landPhoenix · LangfuseApp Insights · LangSmithOTLP fileswrites traces3onlineOpen: Checks we callChecks we callPresidio · Azure AI EvaluationDeepEval · RagasContent Safety · Prompt Guardjudge models · our patternsall tools ›callsBefore release and in batchgolden datasets · CI release gatebackfill on old tracescost rules over days4offlinewrites resultsYour tools, not oursLangfuse / Phoenix scoresApp Insights · Prometheus + GrafanaTeams · ServiceNow alertsContent we shipdashboard templatesalert rules (KQL, PromQL)metric definitions
The numbered circles are where checks plug in: (1) inside the agent, (2) at the gateway, (3) on the traces afterwards, (4) before release or in batch. Dashed boxes are optional.

The four ways in, in plain words

#WhereWhenCan it stop a bad reply?
1Hooks inside the agent's frameworkDuring the turn: inlineYes
2The gateway every model call passes throughDuring the turn: inlineYes, for model calls only
3The traces the agent already writesAfter the turn: onlineNo: it alerts and reports
4Golden test sets, CI, historyBefore release or in batch: offlineIt stops a release, not a reply
What's built today

The engine's check step, the five patterns, four provider checks, and reading traces from a trace file (way 3). The reference agent's own harness already runs checks inline (way 1) using this package. Running any check at the gateway, inline in other frameworks, and writing results back are planned: see every integration.

Decided · The big picture
  1. Does this drawing show what we're trying to do? What would you add or remove before showing it to a client?
    Decided: Yes. For a client, drop the tier labels and keep the four ways in.

Look inside