cleandataengineering.aiEval Harness, drawn simply
You are here: Architecture › One question, three languages

Eval Harness Tutorial · Architecture

One question, three languages

An ops lead asks: “what share of turns failed in the last 15 minutes, for each agent?” On Azure that is a KQL query. On Prometheus it is a PromQL expression. On a warehouse it is SQL. Three teams writing it three times will get three slightly different numbers, and nobody will know which is right.

So we write the question once, as a small recipe called a metric. It says three things and names no tool: what to count, how, and what to split it by. A writer for each language turns the recipe into a query. A test then proves every query gives the number that plain Python gives on the same traces.

One metricwhat to count: turnshow: the share that failedsplit by: agentThe same number?worked out in plain Pythonfrom the same traces:every query must match itKQL writerturns it intoKQLApp Insightsruns the queryPromQL writerturns it intoPromQLPrometheusruns the querySQL writerturns it intoSQLa table of callsruns the queryGreen: the query was run for real and matched. Butter: run in Prometheus's own checker, not yet on a live Prometheus.
The metric is the only thing a person writes. The queries are generated, so they can't drift apart.

The 9 metrics so far

MetricWhat it countsHowSplit byAnswers
turnsturnshow manyagent—
turn_error_rateturnswhat share failedagentA1, A3
model_error_ratemodel callswhat share failedmodelA3
tool_error_ratetool callswhat share failedtoolA3, D1
turn_secondsturnshow long (typical, slow, slowest)agentB1
model_secondsmodel callshow long (typical, slow, slowest)modelB1
tool_secondstool callshow long (typical, slow, slowest)toolD2
time_by_stepmodel calls, tool callstotal timestepB2
slowest_turnsturnsthe slowest few, to open—B3

They cover 7 of the 62 ops questions: the ones about errors and speed. The rest are still to write.

Try it: one metric, three queries

This is turn_error_rate, written by the three writers when this page was built.

KQL, for App Insights

union AppRequests, AppDependencies
| where TimeGenerated > ago(15m)
| extend step = case(tostring(Properties["gen_ai.operation.name"]) == "invoke_agent", "turn", "")
| where step != ""
| extend agent = tostring(Properties["gen_ai.agent.name"]), failed = Success == false
| summarize rate = todouble(countif(failed)) / count() by agent

PromQL, for Prometheus

(sum by (gen_ai_agent_name) (increase(gen_ai_client_operation_duration_seconds_count{gen_ai_operation_name=~"invoke_agent", error_type!=""}[15m])) or sum by (gen_ai_agent_name) (increase(gen_ai_client_operation_duration_seconds_count{gen_ai_operation_name=~"invoke_agent"}[15m])) * 0) / sum by (gen_ai_agent_name) (increase(gen_ai_client_operation_duration_seconds_count{gen_ai_operation_name=~"invoke_agent"}[15m]))

SQL, for a table of call records

select agent, avg(case when failed then 1.0 else 0.0 end) as rate from calls where kind in ('turn') group by agent order by agent

On the recorded traces used in this tutorial (16 steps from a refund agent), all three ways give:

MetricGroupAnswer
turn_error_raterefund-agentrate = 0
tool_error_rateget_orderrate = 0.5
tool_error_rateissue_refundrate = 0
time_by_stepmodelseconds = 0.01981
time_by_steptoolseconds = 0.006535

How we know they agree

LanguageHow it is checkedState
SQLRun in DuckDB on two sets of traces; every metric must equal the referenceBuilt runs in the test suite
PromQLRun in promtool, Prometheus's own checker, on counters added up from the same tracesBuilt runs in the test suite; not yet on a live Prometheus
KQLRun on the claims agent's live App Insights: 404 steps, 9 of 9 metrics the same as the referenceBuilt run on 16 Nov 2026, on test traffic
CloudWatch Logs Insights—Not built yet
What the live run taught us
  • kind is a reserved word in KQL. The first query failed to parse; the column is now called step.
  • App Insights keeps the conversation id in a column of its own (SessionId), not with the other attributes. All 104 turns had it there. Our reader only looked among the attributes, which is why an earlier run reported “no conversation ids”. The reader now reads the column.
  • Azure stores a step's length in whole milliseconds, so sums match to the millisecond, not beyond.

Percentiles from counters are estimates

A detail store keeps every step's time, so it can give the exact middle value. A number store keeps only how many steps fell into each time bucket (up to 1 second, up to 2.5 seconds, and so on), so it can only say which bucket the answer is in. Take four turns that took 2, 4, 9, 1.5 seconds:

Typical (p50)Slow (p95)
From every step's time (detail store)3 s8.25 s
From bucket counts (number store)2.5 s9 s

Counts, rates and totals are exact in both. Only percentiles differ, and a dashboard should say which kind it is showing.

Two things that differ between agents

The recipe stays the same. These two are named in the binding, not assumed:

OpenTelemetry GenAI conventionsThe reference agent harness
A turn isa span whose operation is invoke_agenta span named agent.turn
A model call isoperation chatgen_ai.chat
A tool call isoperation execute_toolagent.tool or agent.tool.local
A turn failed whenthe span has an error statuserror status, or agent.turn.result is failed
The agent's name isgen_ai.agent.namethe service that sent the span
The conversation isgen_ai.conversation.idsession.id

The counters need a name too. The default is the GenAI conventions' duration histogram, gen_ai_client_operation_duration_seconds, labelled by step, agent, model, tool and error.

Not built yet

PieceWhy it matters
A Collector setup that adds up exactly these countersUntil then the PromQL is proven on counters we add up ourselves, not on a real Collector's
A run against a live PrometheusThe last proof for the number store
Writers for CloudWatch and LangfuseThe AWS and open-stack detail stores
“Against normal” for the failure rate (question A1)Today's rate next to the usual rate, so a spike stands out
Metrics for the other 55 questionsCost, safety, changes and usage come next
The Grafana boardBuilt from these same metrics, one panel each
Your review · One question, three languages
  1. The counters are named after the GenAI conventions' duration histogram. Is that the name you want clients to see, or should we use the name the Collector's own span-metrics step produces? My suggestion: keep the conventions' name and let the binding override it.
  2. What comes next: the Grafana board for these nine metrics, or metrics for cost (group C)? My suggestion: the board first, so there is something to show.

Reply in chat with the page name and the number, for example “session 2: …”.

Source: src/evalharness/metrics.py; tests in tests/test_metrics.py; the live run is scripts/metrics_on_app_insights.py, saved in docs/runs/.