Eval Harness Tutorial · Catalog
Ops questions answered from traces
Agent and LLM traces go to LangSmith, Langfuse, App Insights, or as numbers to Prometheus. Whichever tool it is, an ops person asks the same questions of them. This page lists those questions. It is the specification for the saved questions and the ready-made dashboard packs: each one gets built once as a definition and compiled into each tool's own form, as the saved questions already are.
| Count | |
|---|---|
| Questions | 62 |
| Built | 12 |
| Partly built | 7 |
| Not built yet | 43 |
| Need span metadata only (work with content capture off) | 32 |
| Prometheus can answer (numbers about everyone, not one conversation) | 44 |
| Need prompt and response text | 3 |
What a question can need
| Need | Meaning |
|---|---|
meta | Span metadata: times, status, model, token counts, tool name |
content | Prompt and response text (content capture on) |
verdict | Eval or guardrail results written as events |
price | A price table to turn tokens into money |
outcome | A resolved / not resolved signal per task |
version | A release, prompt or model version tag on spans |
feedback | User ratings sent as events |
The trace stores (LangSmith, Langfuse, App Insights) can answer every question. Prometheus holds numbers only, so it answers a question when the answer is a count, rate or trend about all traffic, and never when you need to open one conversation or read its text. This is worked out by one rule in the code, not by hand.
A · Is it working right now?
| # | Question | Needs | Who can answer | State |
|---|---|---|---|---|
| A1 | Is any agent failing right now? What's the error rate in the last 15 minutes against normal? | meta | trace stores + Prometheus | Partly built metric (metrics.py): turn_error_rate; the comparison with normal is not built |
| A2 | Is the model provider slow, down or throttling us (429s, timeouts)? | meta | trace stores + Prometheus | Not built yet |
| A3 | Which agent, tool or model is behind the current errors? | meta | trace stores + Prometheus | Built metric (metrics.py): turn_error_rate, model_error_rate, tool_error_rate |
| A4 | Are requests queuing, with time to first token going up? | meta | trace stores + Prometheus | Not built yet |
| A5 | Did anything start failing right after a deployment? | meta, version | trace stores + Prometheus | Not built yet |
B · Speed
| # | Question | Needs | Who can answer | State |
|---|---|---|---|---|
| B1 | What's p50, p95 and p99 response time, per agent and per model? | meta | trace stores + Prometheus | Built metric (metrics.py): turn_seconds, model_seconds |
| B2 | Where does the time go: model, tools, retrieval, or waiting? | meta | trace stores + Prometheus | Partly built metric (metrics.py): time_by_step; model and tools only |
| B3 | Which conversations took the longest, and why? | meta | trace stores only (one conversation) | Built metric (metrics.py): slowest_turns |
| B4 | How many requests timed out or were cut off? | meta | trace stores + Prometheus | Not built yet |
| B5 | Is time to first token within our target? | meta | trace stores + Prometheus | Not built yet |
C · Cost
| # | Question | Needs | Who can answer | State |
|---|---|---|---|---|
| C1 | What did we spend today and this month, and are we on track against budget? | meta, price | trace stores + Prometheus | Not built yet |
| C2 | What does one resolved task cost, by agent, model, customer and feature? | meta, outcome, price | trace stores + Prometheus | Not built yet |
| C3 | Which 20 conversations cost the most? | meta, price | trace stores only (one conversation) | Not built yet |
| C4 | How much went on waste: retries, loops, failed runs, oversized context? | meta, price | trace stores + Prometheus | Partly built cost rules (costs.py, llm-cost-explainer): LLM-06 bulk on real-time, LLM-22 priority tier |
| C5 | Is token use per conversation creeping up? | meta | trace stores + Prometheus | Not built yet |
| C6 | What's our prompt-cache hit rate, and what did caching save? | meta, price | trace stores + Prometheus | Partly built cost rules (costs.py, llm-cost-explainer): LLM-01 caching off, LLM-26 cache too short |
| C7 | Would a cheaper model do for some of the traffic? | meta, price, verdict | trace stores + Prometheus | Partly built cost rules (costs.py, llm-cost-explainer): LLM-08 traffic moved to a pricier model |
D · Tools and the systems they call
| # | Question | Needs | Who can answer | State |
|---|---|---|---|---|
| D1 | Which tools fail most, and with what error? | meta | trace stores + Prometheus | Partly built metric (metrics.py): tool_error_rate; the kind of error is not split out |
| D2 | Which tools are slowest? | meta | trace stores + Prometheus | Built metric (metrics.py): tool_seconds |
| D3 | How often does the agent call a tool with wrong or invalid arguments? | meta | trace stores + Prometheus | Not built yet |
| D4 | Which outside APIs behind the tools are failing? | meta | trace stores + Prometheus | Not built yet |
| D5 | Where did a tool argument go past its limit? | verdict | trace stores only (one conversation) | Built saved question limits_exceeded |
| D6 | Was any tool called that the agent shouldn't use? | meta | trace stores + Prometheus | Not built yet |
E · Agent behaviour
| # | Question | Needs | Who can answer | State |
|---|---|---|---|---|
| E1 | How many steps does a task take on average, and which ones take far more? | meta | trace stores + Prometheus | Not built yet |
| E2 | Where did the agent loop, calling the same tool again and again? | meta | trace stores only (one conversation) | Not built yet |
| E3 | How many runs hit the step or iteration limit? | meta | trace stores + Prometheus | Not built yet |
| E4 | Which conversations never finished, or were abandoned? | meta | trace stores only (one conversation) | Not built yet |
| E5 | Where should the agent have handed over or asked for approval, and didn't? | verdict | trace stores only (one conversation) | Built saved question under_escalations |
| E6 | Where did it hand over or ask for approval with no rule calling for it? | verdict | trace stores only (one conversation) | Built saved question over_escalations |
| E7 | Where did it act without doing the required step first? | verdict | trace stores only (one conversation) | Built saved question actions_without_prerequisite |
| E8 | How do multiple agents pass work between them, and where does the hand-off break? | meta | trace stores only (one conversation) | Not built yet |
F · Quality and outcomes
| # | Question | Needs | Who can answer | State |
|---|---|---|---|---|
| F1 | What share of tasks were resolved without a human? | meta, outcome | trace stores + Prometheus | Not built yet |
| F2 | What's the user feedback trend (thumbs, ratings)? | feedback | trace stores + Prometheus | Not built yet |
| F3 | Are eval scores holding steady, week on week? | verdict | trace stores + Prometheus | Built weekly report (report.py) |
| F4 | Where did the agent state a fact no system gave it? | verdict | trace stores only (one conversation) | Built saved question unsupported_facts |
| F5 | Where did it say something was done when it wasn't? | verdict | trace stores only (one conversation) | Built saved question promises_not_kept |
| F6 | How often does it refuse or say "I can't help"? | verdict | trace stores + Prometheus | Not built yet |
| F7 | Which kinds of question does it handle worst? | content, verdict | trace stores only | Not built yet |
G · Safety and security
| # | Question | Needs | Who can answer | State |
|---|---|---|---|---|
| G1 | Which guardrails fired, how often, and of what type? | verdict | trace stores + Prometheus | Not built yet |
| G2 | How many of those were false alarms that blocked real work? | feedback, verdict | trace stores + Prometheus | Not built yet |
| G3 | Were there prompt-injection or jailbreak attempts? Did any get through? | verdict | trace stores only (one conversation) | Not built yet |
| G4 | Did personal data appear in prompts, tool outputs or replies? | verdict | trace stores only (one conversation) | Not built yet |
| G5 | Were risky actions (payments, deletes, emails) taken without approval? | meta, verdict | trace stores only (one conversation) | Not built yet |
| G6 | Is any user or key showing unusual volume or behaviour? | meta | trace stores + Prometheus | Not built yet |
H · Changes and regressions
| # | Question | Needs | Who can answer | State |
|---|---|---|---|---|
| H1 | Did the new release, prompt or model change speed, cost or quality? | meta, verdict, version | trace stores + Prometheus | Not built yet |
| H2 | How does version A compare with version B, side by side? | meta, verdict, version | trace stores + Prometheus | Not built yet |
| H3 | Did the provider silently change the model version? | meta | trace stores + Prometheus | Not built yet |
I · Usage
| # | Question | Needs | Who can answer | State |
|---|---|---|---|---|
| I1 | How many users, sessions and requests, per day and per customer? | meta | trace stores + Prometheus | Not built yet |
| I2 | When are the peak hours? | meta | trace stores + Prometheus | Not built yet |
| I3 | What are people asking about most, and are there new topics? | content | trace stores only | Not built yet |
| I4 | Which features or agents are barely used? | meta | trace stores + Prometheus | Not built yet |
J · Models and providers
| # | Question | Needs | Who can answer | State |
|---|---|---|---|---|
| J1 | Which models and versions are in use, and in what share? | meta | trace stores + Prometheus | Partly built cost rules (costs.py, llm-cost-explainer): LLM-24 old model still in use |
| J2 | How often did we fall back to a backup model? | meta | trace stores + Prometheus | Not built yet |
| J3 | How often are we close to the context limit, or cutting input? | meta | trace stores + Prometheus | Not built yet |
| J4 | Why are responses stopping: length, content filter, or normal stop? | meta | trace stores + Prometheus | Not built yet |
K · Can we trust these answers?
| # | Question | Needs | Who can answer | State |
|---|---|---|---|---|
| K1 | What share of traces is missing, dropped or sampled out? | meta | trace stores + Prometheus | Not built yet |
| K2 | Which agents send traces without content, session ids or user ids? | meta | trace stores + Prometheus | Not built yet |
| K3 | What share of conversations could not be judged, and why? | verdict | trace stores + Prometheus | Built weekly report (report.py) |
| K4 | Are spans from the same request linked end to end? | meta | trace stores + Prometheus | Not built yet |
L · Investigating an incident
| # | Question | Needs | Who can answer | State |
|---|---|---|---|---|
| L1 | A customer complained: show me exactly what happened in their session. | content, meta | trace stores only (one conversation) | Not built yet |
| L2 | Find other conversations that failed the same way. | meta, verdict | trace stores only (one conversation) | Not built yet |
| L3 | What changed in the hour before the incident started? | meta, version | trace stores + Prometheus | Not built yet |
- Which 10 should the first dashboard pack answer? My suggestion: A1, A2, B1, C1, C2, D1, E2, F1, G1 and K1, because they need span metadata only and work on every tool.
- Is anything an ops lead asks every week missing from these 12 groups?
Reply in chat with the page name and the number, for example “session 2: …”.
Source: src/evalharness/ops_catalog.py, tested in tests/test_ops_catalog.py. The six built saved questions are in src/evalharness/questions.py.