cleandataengineering.aiEval Harness, drawn simply
You are here: Catalog › Ops questions

Eval Harness Tutorial · Catalog

Ops questions answered from traces

Agent and LLM traces go to LangSmith, Langfuse, App Insights, or as numbers to Prometheus. Whichever tool it is, an ops person asks the same questions of them. This page lists those questions. It is the specification for the saved questions and the ready-made dashboard packs: each one gets built once as a definition and compiled into each tool's own form, as the saved questions already are.

Count
Questions62
Built12
Partly built7
Not built yet43
Need span metadata only (work with content capture off)32
Prometheus can answer (numbers about everyone, not one conversation)44
Need prompt and response text3

What a question can need

NeedMeaning
metaSpan metadata: times, status, model, token counts, tool name
contentPrompt and response text (content capture on)
verdictEval or guardrail results written as events
priceA price table to turn tokens into money
outcomeA resolved / not resolved signal per task
versionA release, prompt or model version tag on spans
feedbackUser ratings sent as events
Who can answer

The trace stores (LangSmith, Langfuse, App Insights) can answer every question. Prometheus holds numbers only, so it answers a question when the answer is a count, rate or trend about all traffic, and never when you need to open one conversation or read its text. This is worked out by one rule in the code, not by hand.

A · Is it working right now?

#QuestionNeedsWho can answerState
A1Is any agent failing right now? What's the error rate in the last 15 minutes against normal?metatrace stores + PrometheusPartly built
metric (metrics.py): turn_error_rate; the comparison with normal is not built
A2Is the model provider slow, down or throttling us (429s, timeouts)?metatrace stores + PrometheusNot built yet
A3Which agent, tool or model is behind the current errors?metatrace stores + PrometheusBuilt
metric (metrics.py): turn_error_rate, model_error_rate, tool_error_rate
A4Are requests queuing, with time to first token going up?metatrace stores + PrometheusNot built yet
A5Did anything start failing right after a deployment?meta, versiontrace stores + PrometheusNot built yet

B · Speed

#QuestionNeedsWho can answerState
B1What's p50, p95 and p99 response time, per agent and per model?metatrace stores + PrometheusBuilt
metric (metrics.py): turn_seconds, model_seconds
B2Where does the time go: model, tools, retrieval, or waiting?metatrace stores + PrometheusPartly built
metric (metrics.py): time_by_step; model and tools only
B3Which conversations took the longest, and why?metatrace stores only (one conversation)Built
metric (metrics.py): slowest_turns
B4How many requests timed out or were cut off?metatrace stores + PrometheusNot built yet
B5Is time to first token within our target?metatrace stores + PrometheusNot built yet

C · Cost

#QuestionNeedsWho can answerState
C1What did we spend today and this month, and are we on track against budget?meta, pricetrace stores + PrometheusNot built yet
C2What does one resolved task cost, by agent, model, customer and feature?meta, outcome, pricetrace stores + PrometheusNot built yet
C3Which 20 conversations cost the most?meta, pricetrace stores only (one conversation)Not built yet
C4How much went on waste: retries, loops, failed runs, oversized context?meta, pricetrace stores + PrometheusPartly built
cost rules (costs.py, llm-cost-explainer): LLM-06 bulk on real-time, LLM-22 priority tier
C5Is token use per conversation creeping up?metatrace stores + PrometheusNot built yet
C6What's our prompt-cache hit rate, and what did caching save?meta, pricetrace stores + PrometheusPartly built
cost rules (costs.py, llm-cost-explainer): LLM-01 caching off, LLM-26 cache too short
C7Would a cheaper model do for some of the traffic?meta, price, verdicttrace stores + PrometheusPartly built
cost rules (costs.py, llm-cost-explainer): LLM-08 traffic moved to a pricier model

D · Tools and the systems they call

#QuestionNeedsWho can answerState
D1Which tools fail most, and with what error?metatrace stores + PrometheusPartly built
metric (metrics.py): tool_error_rate; the kind of error is not split out
D2Which tools are slowest?metatrace stores + PrometheusBuilt
metric (metrics.py): tool_seconds
D3How often does the agent call a tool with wrong or invalid arguments?metatrace stores + PrometheusNot built yet
D4Which outside APIs behind the tools are failing?metatrace stores + PrometheusNot built yet
D5Where did a tool argument go past its limit?verdicttrace stores only (one conversation)Built
saved question limits_exceeded
D6Was any tool called that the agent shouldn't use?metatrace stores + PrometheusNot built yet

E · Agent behaviour

#QuestionNeedsWho can answerState
E1How many steps does a task take on average, and which ones take far more?metatrace stores + PrometheusNot built yet
E2Where did the agent loop, calling the same tool again and again?metatrace stores only (one conversation)Not built yet
E3How many runs hit the step or iteration limit?metatrace stores + PrometheusNot built yet
E4Which conversations never finished, or were abandoned?metatrace stores only (one conversation)Not built yet
E5Where should the agent have handed over or asked for approval, and didn't?verdicttrace stores only (one conversation)Built
saved question under_escalations
E6Where did it hand over or ask for approval with no rule calling for it?verdicttrace stores only (one conversation)Built
saved question over_escalations
E7Where did it act without doing the required step first?verdicttrace stores only (one conversation)Built
saved question actions_without_prerequisite
E8How do multiple agents pass work between them, and where does the hand-off break?metatrace stores only (one conversation)Not built yet

F · Quality and outcomes

#QuestionNeedsWho can answerState
F1What share of tasks were resolved without a human?meta, outcometrace stores + PrometheusNot built yet
F2What's the user feedback trend (thumbs, ratings)?feedbacktrace stores + PrometheusNot built yet
F3Are eval scores holding steady, week on week?verdicttrace stores + PrometheusBuilt
weekly report (report.py)
F4Where did the agent state a fact no system gave it?verdicttrace stores only (one conversation)Built
saved question unsupported_facts
F5Where did it say something was done when it wasn't?verdicttrace stores only (one conversation)Built
saved question promises_not_kept
F6How often does it refuse or say "I can't help"?verdicttrace stores + PrometheusNot built yet
F7Which kinds of question does it handle worst?content, verdicttrace stores onlyNot built yet

G · Safety and security

#QuestionNeedsWho can answerState
G1Which guardrails fired, how often, and of what type?verdicttrace stores + PrometheusNot built yet
G2How many of those were false alarms that blocked real work?feedback, verdicttrace stores + PrometheusNot built yet
G3Were there prompt-injection or jailbreak attempts? Did any get through?verdicttrace stores only (one conversation)Not built yet
G4Did personal data appear in prompts, tool outputs or replies?verdicttrace stores only (one conversation)Not built yet
G5Were risky actions (payments, deletes, emails) taken without approval?meta, verdicttrace stores only (one conversation)Not built yet
G6Is any user or key showing unusual volume or behaviour?metatrace stores + PrometheusNot built yet

H · Changes and regressions

#QuestionNeedsWho can answerState
H1Did the new release, prompt or model change speed, cost or quality?meta, verdict, versiontrace stores + PrometheusNot built yet
H2How does version A compare with version B, side by side?meta, verdict, versiontrace stores + PrometheusNot built yet
H3Did the provider silently change the model version?metatrace stores + PrometheusNot built yet

I · Usage

#QuestionNeedsWho can answerState
I1How many users, sessions and requests, per day and per customer?metatrace stores + PrometheusNot built yet
I2When are the peak hours?metatrace stores + PrometheusNot built yet
I3What are people asking about most, and are there new topics?contenttrace stores onlyNot built yet
I4Which features or agents are barely used?metatrace stores + PrometheusNot built yet

J · Models and providers

#QuestionNeedsWho can answerState
J1Which models and versions are in use, and in what share?metatrace stores + PrometheusPartly built
cost rules (costs.py, llm-cost-explainer): LLM-24 old model still in use
J2How often did we fall back to a backup model?metatrace stores + PrometheusNot built yet
J3How often are we close to the context limit, or cutting input?metatrace stores + PrometheusNot built yet
J4Why are responses stopping: length, content filter, or normal stop?metatrace stores + PrometheusNot built yet

K · Can we trust these answers?

#QuestionNeedsWho can answerState
K1What share of traces is missing, dropped or sampled out?metatrace stores + PrometheusNot built yet
K2Which agents send traces without content, session ids or user ids?metatrace stores + PrometheusNot built yet
K3What share of conversations could not be judged, and why?verdicttrace stores + PrometheusBuilt
weekly report (report.py)
K4Are spans from the same request linked end to end?metatrace stores + PrometheusNot built yet

L · Investigating an incident

#QuestionNeedsWho can answerState
L1A customer complained: show me exactly what happened in their session.content, metatrace stores only (one conversation)Not built yet
L2Find other conversations that failed the same way.meta, verdicttrace stores only (one conversation)Not built yet
L3What changed in the hour before the incident started?meta, versiontrace stores + PrometheusNot built yet
Your review · Ops questions
  1. Which 10 should the first dashboard pack answer? My suggestion: A1, A2, B1, C1, C2, D1, E2, F1, G1 and K1, because they need span metadata only and work on every tool.
  2. Is anything an ops lead asks every week missing from these 12 groups?

Reply in chat with the page name and the number, for example “session 2: …”.

Source: src/evalharness/ops_catalog.py, tested in tests/test_ops_catalog.py. The six built saved questions are in src/evalharness/questions.py.