Eval Harness Tutorial · Architecture
The whole chain: from agent code to a dashboard
An agent does its work, writes down what it did, and that record travels to a tool where people look at it. The eval harness sits along that road. This page draws the whole road once, top to bottom, so you can see where each part of the framework stands and which parts are tools you already have.
Read the colours as a status report. Blue is a tool that already exists and isn't ours. Green is ours and built. A dashed white box is ours and not built yet.
The five rows, in plain words
| Row | What happens there | Same on every platform? | Our part |
|---|---|---|---|
| 1 · Agent harness | The agent loops: ask the model, call a tool, repeat, reply. A check can step in at five moments. Every step is written down as a span. | Yes | Inline checks: built. The loop and the telemetry are the agent's own. |
| 2 · What leaves the agent | Spans in the OpenTelemetry GenAI format, plus two kinds of event from us: what a rule decided, and what a check found. | Yes. This is the first fixed contract. | Verdict events and decision events: built. |
| 3 · Where it lands | The client's trace store. Beside it runs the eval worker: it reads the traces, rebuilds each conversation, runs the rules and writes verdicts back into the same store. | No. Each store has its own query language. | One reader per store. App Insights is run live; CloudWatch and Phoenix are built and wait for a live run; the rest are not built. |
| 4 · The common questions | One list of what ops asks of any agent. Each question becomes a definition, and a small query writer turns it into the store's language. | The list and the definitions, yes. The query writer, no. | 62 questions: 12 answered, 7 partly, 43 not yet. 9 metric definitions are written, with query writers for KQL, PromQL and SQL. |
| 5 · What people see | The client's own screens. We ship what goes on them: a report, a dashboard template, alert rules. | The content, yes. The screen is theirs. | Weekly report: built. Dashboard templates and alerts: not built. |
Prometheus sits in row 3 as an optional second store for totals. Some platforms keep detail and totals in one store, others in two: see one store or two.
The middle of the chain can't be generic, because Azure, AWS and Databricks each store traces their own way. So we fix the two ends instead. Row 2 fixes what goes in. Row 4 fixes what is asked. Between them, a platform needs only a reader, a writer and a query writer.
What every agent gets, and what each agent still writes
The framework does the parts that are the same for every agent. A team adds only what the framework can't know: the agent's own tools, limits and rules.
| Comes with the framework | State |
|---|---|
| The trace and verdict formats, and rebuilding conversations from traces | Built |
| General checks: personal data, groundedness, tool use, as a ready-made preset | Built |
| The logic of the five rule patterns (the team only fills in names and numbers) | Built |
| Rule modes (watch, alert, block), locked rules, the audit list | Built |
| Cost rules | 6 of 36 built |
| Ops questions that need span metadata only: 32 of 62, with no work per agent once their queries exist | List written; queries not built |
| Weekly report | Built |
| Dashboard templates and alert rules | Not built |
| Each agent's team still does | How much |
|---|---|
| Turn on tracing, with a conversation id on every span | Once, a few lines; no work if the framework already traces |
| Write its rules file: its tool names, its limits, when to hand over to a person | The claims agent's file is 62 lines and holds 5 rules of its own (read it) |
| Name its own judge model | One setting |
| Say how a conversation ended (resolved, handed over, abandoned) | One attribute on the trace |
| Name an owner per rule, and label about 50 conversations before a rule may block | A few hours per blocking rule |
- Is this the drawing you would show a CTO, or should the client version hide the status colours and show only the five rows?
- Row 3 lists eight stores. Which three matter most for the clients you expect first? My suggestion: App Insights, Databricks MLflow traces, then CloudWatch.
Reply in chat with the page name and the number, for example “session 2: …”.
Counts are read from src/evalharness/ops_catalog.py and examples/claims-fnol.yaml when this page is built.