Eval Harness Tutorial · Architecture
Where the signals go: one store or two
Think of a shop. Every sale prints a receipt: what was sold, when, to whom. The till also keeps a running total for the day. You can add up the receipts to get the total, but you can never get a receipt back out of a total. And if you throw away nine receipts in ten to save space, adding up what's left gives the wrong total.
An agent's telemetry works the same way:
- A span is a receipt: the full record of one model call or tool call.
- A counter is the till total: how many calls, how many failed, how long they took.
- An event is a note pinned to a receipt. Our verdicts and decisions are events.
Some platforms keep receipts and totals in one place. Others keep them in two. The eval harness must work the same either way, so it must not be designed around either one.
The roles a platform fills
| Role | What it keeps | What it answers | Azure today | Open stack today | Databricks (planned) |
|---|---|---|---|---|---|
| Detail store | Every span and event | Open one conversation; what happened, in order; why a rule failed | App Insights | Langfuse | MLflow traces |
| Number store | Counters | Rates, totals, response times, alerts that run every minute | App Insights, counted from the spans when asked | Prometheus | Tables, counted with SQL |
| History store | Call records kept for months | Cost rules over days, long trends | Log Analytics | DuckDB or ClickHouse | Delta tables |
One tool may fill two or three roles. That is the whole difference between shape A and shape B, and it lives in the binding file, not in the checks.
Five rules that keep it flexible
- The agent sends once and never names a store. It writes spans, counters and events in the OpenTelemetry format. What happens next is routing.
- The binding names a tool for each role and says what that tool can do: does it keep every span or a sample, does it hold the customer's words, how long does it keep data, how fresh is it.
- Each question says what it needs, not which tool answers it. A planner sends it to the cheapest role that can answer it on this platform. If no role can, the answer is “can't be answered here”, never a wrong number.
- Every verdict is written twice from one call: as an event on the trace, so you can open the conversation, and as a counter, so you can count and alert cheaply.
- Totals are added up before anything is thrown away. If traces are sampled, counters come from the agent or the Collector, never from the sample. A check that looks at only a share of conversations records that share on its verdicts.
Of the 62 ops questions, 44 are about totals and 18 need to open a conversation. On Azure today both kinds go to App Insights. On the open stack the first kind goes to Prometheus and the second to Langfuse. The question is written once; only the route changes.
Which shapes we support
| Shape | Example | Verdict |
|---|---|---|
| A · One store does both, keeps every span | App Insights today | Pursue. The default on Azure: nothing extra to run. |
| B · Detail store plus number store | Langfuse plus Prometheus; App Insights plus managed Prometheus | Pursue. Add the number store when alerts must run more often than every 15 minutes, when traces are sampled, or when many agents share one board. |
| One store, but it keeps only a sample | Any trace store with sampling switched on | Pursue with a guard. Questions about totals are refused unless a number store exists or the sample share is recorded. |
| Number store only, no traces | Prometheus alone | Limited. Totals only. No conversation can be opened and no rule about what the agent did can run. |
| Lakehouse | MLflow traces and Delta tables on Databricks | Pursue later. One SQL store fills all three roles. |
What the binding files say
These two ship with the package. keeps is all or sample;
words says whether the conversation's text is stored; from says where the totals are added up.
# stacks/azure.yaml stores: detail: {tool: app_insights, keeps: all, words: false} numbers: {tool: app_insights, from: spans} history: {tool: log_analytics} # stacks/open.yaml stores: detail: {tool: langfuse, keeps: all, words: true} numbers: {tool: prometheus, from: collector} history: {tool: duckdb}
Try it: the same 62 questions on two platforms
The planner read both files when this page was built and routed every question:
| Platform | Totals go to | Conversations go to | Can't be answered |
|---|---|---|---|
| Azure (shape A) | 44 → app_insights | 15 → app_insights | 3: F7, I3, L1 |
| Open stack (shape B) | 44 → prometheus | 18 → langfuse | 0 |
| Question | Platform | Goes to | Why |
|---|---|---|---|
| B1 · What's p50, p95 and p99 response time, per agent and per model? | Azure (shape A) | numbers store (app_insights) | counted from the spans when asked |
| B1 · What's p50, p95 and p99 response time, per agent and per model? | Open stack (shape B) | numbers store (prometheus) | from counters |
| E5 · Where should the agent have handed over or asked for approval, and didn't? | Azure (shape A) | detail store (app_insights) | opens the conversations |
| E5 · Where should the agent have handed over or asked for approval, and didn't? | Open stack (shape B) | detail store (langfuse) | opens the conversations |
| I3 · What are people asking about most, and are there new topics? | Azure (shape A) | can't be answered here | the detail store doesn't keep the words |
| I3 · What are people asking about most, and are there new topics? | Open stack (shape B) | detail store (langfuse) | opens the conversations |
Azure can't answer the three questions that read the customer's words, because production keeps the words out of the store (decision T30). That is the honest answer, and the planner gives it instead of an empty chart.
What this changed in the code
| Piece | Before | Now | State |
|---|---|---|---|
| Binding file | Named an adapter for each port | stores: a tool for each role, and what it can do. A binding that can't be true is refused when loaded. | Built |
| Ops question | Listed tools by name | Says whether it is a total or opens conversations; the planner picks the role | Built |
| Verdict writer | Events only | An event and a counter (evalharness.verdicts) from one call | Built |
| Verdict record | Said nothing about sampling | The event carries the share of conversations that were checked | Built |
| Counter on a real number store | — | The counter is tested in memory; it hasn't been sent to a live Prometheus or App Insights yet | Not built yet |
| Same-answer test | None | One recorded trace through a detail store and a number store must give the same number | Not built yet |
- Should a binding name stores by role (detail, numbers, history), as drawn?
Decided: Yes (T31). - Who adds up the counters?
Decided: The Collector where there is one; the store at question time when it is a single store that keeps every span; never from a sample (T32). - Should every verdict be written as an event and a counter from one call?
Decided: Yes (T33). - Which shape do we present first to an Azure client?
Decided: Shape A, App Insights alone, with shape B as the stated next step and its three triggers (T34).
Source: src/evalharness/stores.py, src/evalharness/stacks/, src/evalharness/sinks/otel_counters.py; tested in tests/test_stores.py and tests/test_sinks_counters.py. See the whole chain for where the stores sit.