Agent Harness Tutorial · End to end · Zoom level 3
Dashboards
One dashboard, four rows, each answering one question for one audience: is it working? (operations), is it doing its job? (delivery), is it right? (testers and risk), what does it cost? (the business). The drawing below is laid out from the real dashboard file, panel by panel.
| Row | Panel | Type | Query (PromQL) |
|---|---|---|---|
| Is it working? | Canary failures, last 30 min | stat | sum(increase(agent_canary_cases_total{outcome="fail"}[30m])) or vector(0) |
| Is it working? | Failed turns, last hour | stat | sum(increase(agent_turns_total{synthetic="false",result="failed"}[1h])) or vector(0) |
| Is it working? | p95 turn latency | stat | histogram_quantile(0.95, sum by (le) (rate(agent_turn_duration_seconds_bucket{synthetic="false"}[15m]))) |
| Is it working? | Alerts firing | stat | count(ALERTS{alertstate="firing"}) or vector(0) |
| Is it working? | Turns by how they ended | timeseries | sum by (result) (rate(agent_turns_total{synthetic="false"}[5m])) * 60 |
| Is it working? | Latency: turn, model call, tool (p95) | timeseries | histogram_quantile(0.95, sum by (le) (rate(agent_turn_duration_seconds_bucket{synthetic="false"}[5m]))); histogram_quantile(0.95, sum by (le) (rate(gen_ai_client_operation_duration_seconds_bucket[5m]))); histogram_quantile(0.95, sum by (tool, le) (rate(agent_tool_duration_seconds_bucket[5m]))) |
| Is it doing its job? | Escalations by rule (share of turns) | timeseries | sum by (rule) (rate(agent_escalations_total{synthetic="false"}[15m])) / scalar(sum(rate(agent_turns_total{synthetic="false"}[15m]))) |
| Is it doing its job? | Refusals by rule (share of turns) | timeseries | sum by (rule) (rate(agent_refusals_total{synthetic="false"}[15m])) / scalar(sum(rate(agent_turns_total{synthetic="false"}[15m]))) |
| Is it doing its job? | Tool calls by outcome | timeseries | sum by (tool, outcome) (rate(agent_tool_calls_total[5m])) * 60 |
| Is it doing its job? | Why runs stopped | timeseries | sum by (reason) (rate(agent_run_terminations_total[15m])) * 60 |
| Is it doing its job? | Model calls by outcome | timeseries | sum by (outcome) (rate(agent_model_calls_total[5m])) * 60 |
| Is it doing its job? | Waits: approvals settled, escalations closed | timeseries | sum by (outcome) (increase(agent_approvals_settled_total[1h])); sum by (state) (increase(agent_escalations_closed_total[1h])) |
| Is it right? | Online findings by rule | timeseries | sum by (rule, severity) (increase(agent_online_findings_total[1h])) |
| Is it right? | Outcomes after the turn | timeseries | sum by (kind) (increase(agent_outcomes_total[1d])) |
| Is it right? | Came back within a day (share) | stat | agent:returned_share:1d |
| Is it right? | Asked for a person after an answer | stat | agent:asked_for_person_share:1d |
| Is it right? | Thumbs down (share of feedback) | stat | agent:feedback_down_share:1d |
| Is it right? | Turns evaluated, today | stat | sum(increase(agent_online_evaluated_total[1d])) or vector(0) |
| What does it cost? | Cost per run (p50, p95) | timeseries | histogram_quantile(0.5, sum by (le) (rate(agent_spend_USD_bucket[1h]))); histogram_quantile(0.95, sum by (le) (rate(agent_spend_USD_bucket[1h]))) |
| What does it cost? | Tokens per minute | timeseries | sum by (gen_ai_token_type) (rate(gen_ai_client_token_usage_sum[5m])) * 60 |
| What does it cost? | Tool calls per turn | timeseries | agent:tool_calls_per_turn:1d |