Eval Harness Tutorial · Architecture
One question, three languages
An ops lead asks: “what share of turns failed in the last 15 minutes, for each agent?” On Azure that is a KQL query. On Prometheus it is a PromQL expression. On a warehouse it is SQL. Three teams writing it three times will get three slightly different numbers, and nobody will know which is right.
So we write the question once, as a small recipe called a metric. It says three things and names no tool: what to count, how, and what to split it by. A writer for each language turns the recipe into a query. A test then proves every query gives the number that plain Python gives on the same traces.
The 9 metrics so far
| Metric | What it counts | How | Split by | Answers |
|---|---|---|---|---|
turns | turns | how many | agent | — |
turn_error_rate | turns | what share failed | agent | A1, A3 |
model_error_rate | model calls | what share failed | model | A3 |
tool_error_rate | tool calls | what share failed | tool | A3, D1 |
turn_seconds | turns | how long (typical, slow, slowest) | agent | B1 |
model_seconds | model calls | how long (typical, slow, slowest) | model | B1 |
tool_seconds | tool calls | how long (typical, slow, slowest) | tool | D2 |
time_by_step | model calls, tool calls | total time | step | B2 |
slowest_turns | turns | the slowest few, to open | — | B3 |
They cover 7 of the 62 ops questions: the ones about errors and speed. The rest are still to write.
Try it: one metric, three queries
This is turn_error_rate, written by the three writers when this page was built.
KQL, for App Insights
union AppRequests, AppDependencies | where TimeGenerated > ago(15m) | extend step = case(tostring(Properties["gen_ai.operation.name"]) == "invoke_agent", "turn", "") | where step != "" | extend agent = tostring(Properties["gen_ai.agent.name"]), failed = Success == false | summarize rate = todouble(countif(failed)) / count() by agent
PromQL, for Prometheus
(sum by (gen_ai_agent_name) (increase(gen_ai_client_operation_duration_seconds_count{gen_ai_operation_name=~"invoke_agent", error_type!=""}[15m])) or sum by (gen_ai_agent_name) (increase(gen_ai_client_operation_duration_seconds_count{gen_ai_operation_name=~"invoke_agent"}[15m])) * 0) / sum by (gen_ai_agent_name) (increase(gen_ai_client_operation_duration_seconds_count{gen_ai_operation_name=~"invoke_agent"}[15m]))
SQL, for a table of call records
select agent, avg(case when failed then 1.0 else 0.0 end) as rate from calls where kind in ('turn') group by agent order by agent
On the recorded traces used in this tutorial (16 steps from a refund agent), all three ways give:
| Metric | Group | Answer |
|---|---|---|
turn_error_rate | refund-agent | rate = 0 |
tool_error_rate | get_order | rate = 0.5 |
tool_error_rate | issue_refund | rate = 0 |
time_by_step | model | seconds = 0.01981 |
time_by_step | tool | seconds = 0.006535 |
How we know they agree
| Language | How it is checked | State |
|---|---|---|
| SQL | Run in DuckDB on two sets of traces; every metric must equal the reference | Built runs in the test suite |
| PromQL | Run in promtool, Prometheus's own checker, on counters added up from the same traces | Built runs in the test suite; not yet on a live Prometheus |
| KQL | Run on the claims agent's live App Insights: 404 steps, 9 of 9 metrics the same as the reference | Built run on 16 Nov 2026, on test traffic |
| CloudWatch Logs Insights | — | Not built yet |
kindis a reserved word in KQL. The first query failed to parse; the column is now calledstep.- App Insights keeps the conversation id in a column of its own (
SessionId), not with the other attributes. All 104 turns had it there. Our reader only looked among the attributes, which is why an earlier run reported “no conversation ids”. The reader now reads the column. - Azure stores a step's length in whole milliseconds, so sums match to the millisecond, not beyond.
Percentiles from counters are estimates
A detail store keeps every step's time, so it can give the exact middle value. A number store keeps only how many steps fell into each time bucket (up to 1 second, up to 2.5 seconds, and so on), so it can only say which bucket the answer is in. Take four turns that took 2, 4, 9, 1.5 seconds:
| Typical (p50) | Slow (p95) | |
|---|---|---|
| From every step's time (detail store) | 3 s | 8.25 s |
| From bucket counts (number store) | 2.5 s | 9 s |
Counts, rates and totals are exact in both. Only percentiles differ, and a dashboard should say which kind it is showing.
Two things that differ between agents
The recipe stays the same. These two are named in the binding, not assumed:
| OpenTelemetry GenAI conventions | The reference agent harness | |
|---|---|---|
| A turn is | a span whose operation is invoke_agent | a span named agent.turn |
| A model call is | operation chat | gen_ai.chat |
| A tool call is | operation execute_tool | agent.tool or agent.tool.local |
| A turn failed when | the span has an error status | error status, or agent.turn.result is failed |
| The agent's name is | gen_ai.agent.name | the service that sent the span |
| The conversation is | gen_ai.conversation.id | session.id |
The counters need a name too. The default is the GenAI conventions' duration
histogram, gen_ai_client_operation_duration_seconds, labelled by step, agent, model, tool and error.
Not built yet
| Piece | Why it matters |
|---|---|
| A Collector setup that adds up exactly these counters | Until then the PromQL is proven on counters we add up ourselves, not on a real Collector's |
| A run against a live Prometheus | The last proof for the number store |
| Writers for CloudWatch and Langfuse | The AWS and open-stack detail stores |
| “Against normal” for the failure rate (question A1) | Today's rate next to the usual rate, so a spike stands out |
| Metrics for the other 55 questions | Cost, safety, changes and usage come next |
| The Grafana board | Built from these same metrics, one panel each |
- The counters are named after the GenAI conventions' duration histogram. Is that the name you want clients to see, or should we use the name the Collector's own span-metrics step produces? My suggestion: keep the conventions' name and let the binding override it.
- What comes next: the Grafana board for these nine metrics, or metrics for cost (group C)? My suggestion: the board first, so there is something to show.
Reply in chat with the page name and the number, for example “session 2: …”.
Source: src/evalharness/metrics.py; tests in tests/test_metrics.py; the live run is scripts/metrics_on_app_insights.py, saved in docs/runs/.