cleandataengineering.aiEval Harness, drawn simply
You are here: Architecture › One store or two

Eval Harness Tutorial · Architecture

Where the signals go: one store or two

Think of a shop. Every sale prints a receipt: what was sold, when, to whom. The till also keeps a running total for the day. You can add up the receipts to get the total, but you can never get a receipt back out of a total. And if you throw away nine receipts in ten to save space, adding up what's left gives the wrong total.

An agent's telemetry works the same way:

  • A span is a receipt: the full record of one model call or tool call.
  • A counter is the till total: how many calls, how many failed, how long they took.
  • An event is a note pinned to a receipt. Our verdicts and decisions are events.

Some platforms keep receipts and totals in one place. Others keep them in two. The eval harness must work the same either way, so it must not be designed around either one.

A · ONE STORE DOES BOTHAzure, as built and deployed todayAgentsends onceAzure Monitor exporterinside the agentspans and eventsApp Insightsevery step, and the totalscounted from the stepsB · TWO STORESOpen stack, as built todayAgentsends onceCollectorsplits by kind of signalspans, eventscountersLangfusekeeps every stepPrometheuskeeps the totalsC · TWO STORES ON AZUREPossible, not builtAgentsends onceCollectorsplits by kind of signalspans, eventscountersApp Insightskeeps every stepManaged Prometheuskeeps the totalsThe eval harness asks for a role, never a tooldetail store: open one conversation, step by step · number store: cheap totals with nothing left outthe binding file says which tool plays each role on this platform
The agent sends once in all three. Only the routing below it differs. The eval harness names roles, and the binding file fills them in.

The roles a platform fills

RoleWhat it keepsWhat it answersAzure todayOpen stack todayDatabricks (planned)
Detail storeEvery span and eventOpen one conversation; what happened, in order; why a rule failedApp InsightsLangfuseMLflow traces
Number storeCountersRates, totals, response times, alerts that run every minuteApp Insights, counted from the spans when askedPrometheusTables, counted with SQL
History storeCall records kept for monthsCost rules over days, long trendsLog AnalyticsDuckDB or ClickHouseDelta tables

One tool may fill two or three roles. That is the whole difference between shape A and shape B, and it lives in the binding file, not in the checks.

Five rules that keep it flexible

  1. The agent sends once and never names a store. It writes spans, counters and events in the OpenTelemetry format. What happens next is routing.
  2. The binding names a tool for each role and says what that tool can do: does it keep every span or a sample, does it hold the customer's words, how long does it keep data, how fresh is it.
  3. Each question says what it needs, not which tool answers it. A planner sends it to the cheapest role that can answer it on this platform. If no role can, the answer is “can't be answered here”, never a wrong number.
  4. Every verdict is written twice from one call: as an event on the trace, so you can open the conversation, and as a counter, so you can count and alert cheaply.
  5. Totals are added up before anything is thrown away. If traces are sampled, counters come from the agent or the Collector, never from the sample. A check that looks at only a share of conversations records that share on its verdicts.
Why rule 3 matters most

Of the 62 ops questions, 44 are about totals and 18 need to open a conversation. On Azure today both kinds go to App Insights. On the open stack the first kind goes to Prometheus and the second to Langfuse. The question is written once; only the route changes.

Which shapes we support

ShapeExampleVerdict
A · One store does both, keeps every spanApp Insights todayPursue. The default on Azure: nothing extra to run.
B · Detail store plus number storeLangfuse plus Prometheus; App Insights plus managed PrometheusPursue. Add the number store when alerts must run more often than every 15 minutes, when traces are sampled, or when many agents share one board.
One store, but it keeps only a sampleAny trace store with sampling switched onPursue with a guard. Questions about totals are refused unless a number store exists or the sample share is recorded.
Number store only, no tracesPrometheus aloneLimited. Totals only. No conversation can be opened and no rule about what the agent did can run.
LakehouseMLflow traces and Delta tables on DatabricksPursue later. One SQL store fills all three roles.

What the binding files say

These two ship with the package. keeps is all or sample; words says whether the conversation's text is stored; from says where the totals are added up.

# stacks/azure.yaml
stores:
  detail:  {tool: app_insights, keeps: all, words: false}
  numbers: {tool: app_insights, from: spans}
  history: {tool: log_analytics}

# stacks/open.yaml
stores:
  detail:  {tool: langfuse, keeps: all, words: true}
  numbers: {tool: prometheus, from: collector}
  history: {tool: duckdb}

Try it: the same 62 questions on two platforms

The planner read both files when this page was built and routed every question:

PlatformTotals go toConversations go toCan't be answered
Azure (shape A)44 → app_insights15 → app_insights3: F7, I3, L1
Open stack (shape B)44 → prometheus18 → langfuse0
QuestionPlatformGoes toWhy
B1 · What's p50, p95 and p99 response time, per agent and per model?Azure (shape A)numbers store (app_insights)counted from the spans when asked
B1 · What's p50, p95 and p99 response time, per agent and per model?Open stack (shape B)numbers store (prometheus)from counters
E5 · Where should the agent have handed over or asked for approval, and didn't?Azure (shape A)detail store (app_insights)opens the conversations
E5 · Where should the agent have handed over or asked for approval, and didn't?Open stack (shape B)detail store (langfuse)opens the conversations
I3 · What are people asking about most, and are there new topics?Azure (shape A)can't be answered herethe detail store doesn't keep the words
I3 · What are people asking about most, and are there new topics?Open stack (shape B)detail store (langfuse)opens the conversations

Azure can't answer the three questions that read the customer's words, because production keeps the words out of the store (decision T30). That is the honest answer, and the planner gives it instead of an empty chart.

What this changed in the code

PieceBeforeNowState
Binding fileNamed an adapter for each portstores: a tool for each role, and what it can do. A binding that can't be true is refused when loaded.Built
Ops questionListed tools by nameSays whether it is a total or opens conversations; the planner picks the roleBuilt
Verdict writerEvents onlyAn event and a counter (evalharness.verdicts) from one callBuilt
Verdict recordSaid nothing about samplingThe event carries the share of conversations that were checkedBuilt
Counter on a real number store—The counter is tested in memory; it hasn't been sent to a live Prometheus or App Insights yetNot built yet
Same-answer testNoneOne recorded trace through a detail store and a number store must give the same numberNot built yet
Decided · One store or two
  1. Should a binding name stores by role (detail, numbers, history), as drawn?
    Decided: Yes (T31).
  2. Who adds up the counters?
    Decided: The Collector where there is one; the store at question time when it is a single store that keeps every span; never from a sample (T32).
  3. Should every verdict be written as an event and a counter from one call?
    Decided: Yes (T33).
  4. Which shape do we present first to an Azure client?
    Decided: Shape A, App Insights alone, with shape B as the stated next step and its three triggers (T34).

Source: src/evalharness/stores.py, src/evalharness/stacks/, src/evalharness/sinks/otel_counters.py; tested in tests/test_stores.py and tests/test_sinks_counters.py. See the whole chain for where the stores sit.