Eval Harness Tutorial · Architecture
Where the framework fits
This is what we're trying to do, in one drawing. Your agent and the tools around it stay as they are. The framework (the dark box) plugs in at four points, numbered below, and writes its verdicts back into tools your team already uses. It has no database, dashboard or chat window of its own.
The four ways in, in plain words
| # | Where | When | Can it stop a bad reply? |
|---|---|---|---|
| 1 | Hooks inside the agent's framework | During the turn: inline | Yes |
| 2 | The gateway every model call passes through | During the turn: inline | Yes, for model calls only |
| 3 | The traces the agent already writes | After the turn: online | No: it alerts and reports |
| 4 | Golden test sets, CI, history | Before release or in batch: offline | It stops a release, not a reply |
What's built today
The engine's check step, the five patterns, four provider checks, and reading traces from a trace file (way 3). The reference agent's own harness already runs checks inline (way 1) using this package. Running any check at the gateway, inline in other frameworks, and writing results back are planned: see every integration.
Decided · The big picture
- Does this drawing show what we're trying to do? What would you add or remove before showing it to a client?
Decided: Yes. For a client, drop the tier labels and keep the four ways in.
Look inside
When checks run: offline, inline, onlineThree timings, what each can do, and every point a check plugs in.Every tool we read from, call, or write toOpen source and cloud tools, upstream and downstream, with what's built.Inside the engine: ports and adaptersNine ports: the fixed shapes every tool plugs into.The whole chain: from agent code to a dashboardOne drawing of every part, from the agent's loop to the screen, coloured by what is built.Where the signals go: one store or twoSome platforms keep everything in one store, others split detail and totals. The design that works for both.One question, three languagesA question is written once as a metric, then asked in each store's own language and checked to give the same number.Hooks in a loop of your ownFive lines in your loop, and a matcher on each hook to target one tool.Who may change a rule, and when it may blockThree layers of rules files, locked rules, an audit list, and the promotion gate.Deciding, masking, and counting across conversationsThe decide step, masking PII instead of blocking, and per-customer limits.The second engine: cost rulesRules over a day of calls, not one conversation, from llm-cost-explainer.What you get on top of your toolsResults, rules, dashboards, conversation and saved questions, tool by tool.