cleandataengineering.aiAgent Harness, drawn simply
You are here: End to end › Eval › Online › Dashboards

Agent Harness Tutorial · End to end · Zoom level 3

Dashboards

One dashboard, four rows, each answering one question for one audience: is it working? (operations), is it doing its job? (delivery), is it right? (testers and risk), what does it cost? (the business). The drawing below is laid out from the real dashboard file, panel by panel.

Is it working? — L0 up, L1 speed and errors, L3 canaryCanary failures, last 30 min123Failed turns, last hour123p95 turn latency123Alerts firing123Turns by how they endedLatency: turn, model call, tool (p95)Is it doing its job? — L2 behaviour ratesEscalations by rule (share of turns)Refusals by rule (share of turns)Tool calls by outcomeWhy runs stoppedModel calls by outcomeWaits: approvals settled, escalations closedIs it right? — L4 online rules and outcomesOnline findings by ruleOutcomes after the turnCame back within a day (share)123Asked for a person after an answ…123Thumbs down (share of feedback)123Turns evaluated, today123What does it cost?Cost per run (p50, p95)Tokens per minuteTool calls per turn
"Support agent — is it working, is it right": the Grafana dashboard in the reference agent (deploy/grafana/dashboards/agent.json). On Azure, Managed Grafana and an Azure Workbook are not built yet; today the same questions are KQL in Log Analytics.
RowPanelTypeQuery (PromQL)
Is it working?Canary failures, last 30 minstatsum(increase(agent_canary_cases_total{outcome="fail"}[30m])) or vector(0)
Is it working?Failed turns, last hourstatsum(increase(agent_turns_total{synthetic="false",result="failed"}[1h])) or vector(0)
Is it working?p95 turn latencystathistogram_quantile(0.95, sum by (le) (rate(agent_turn_duration_seconds_bucket{synthetic="false"}[15m])))
Is it working?Alerts firingstatcount(ALERTS{alertstate="firing"}) or vector(0)
Is it working?Turns by how they endedtimeseriessum by (result) (rate(agent_turns_total{synthetic="false"}[5m])) * 60
Is it working?Latency: turn, model call, tool (p95)timeserieshistogram_quantile(0.95, sum by (le) (rate(agent_turn_duration_seconds_bucket{synthetic="false"}[5m]))); histogram_quantile(0.95, sum by (le) (rate(gen_ai_client_operation_duration_seconds_bucket[5m]))); histogram_quantile(0.95, sum by (tool, le) (rate(agent_tool_duration_seconds_bucket[5m])))
Is it doing its job?Escalations by rule (share of turns)timeseriessum by (rule) (rate(agent_escalations_total{synthetic="false"}[15m])) / scalar(sum(rate(agent_turns_total{synthetic="false"}[15m])))
Is it doing its job?Refusals by rule (share of turns)timeseriessum by (rule) (rate(agent_refusals_total{synthetic="false"}[15m])) / scalar(sum(rate(agent_turns_total{synthetic="false"}[15m])))
Is it doing its job?Tool calls by outcometimeseriessum by (tool, outcome) (rate(agent_tool_calls_total[5m])) * 60
Is it doing its job?Why runs stoppedtimeseriessum by (reason) (rate(agent_run_terminations_total[15m])) * 60
Is it doing its job?Model calls by outcometimeseriessum by (outcome) (rate(agent_model_calls_total[5m])) * 60
Is it doing its job?Waits: approvals settled, escalations closedtimeseriessum by (outcome) (increase(agent_approvals_settled_total[1h])); sum by (state) (increase(agent_escalations_closed_total[1h]))
Is it right?Online findings by ruletimeseriessum by (rule, severity) (increase(agent_online_findings_total[1h]))
Is it right?Outcomes after the turntimeseriessum by (kind) (increase(agent_outcomes_total[1d]))
Is it right?Came back within a day (share)statagent:returned_share:1d
Is it right?Asked for a person after an answerstatagent:asked_for_person_share:1d
Is it right?Thumbs down (share of feedback)statagent:feedback_down_share:1d
Is it right?Turns evaluated, todaystatsum(increase(agent_online_evaluated_total[1d])) or vector(0)
What does it cost?Cost per run (p50, p95)timeserieshistogram_quantile(0.5, sum by (le) (rate(agent_spend_USD_bucket[1h]))); histogram_quantile(0.95, sum by (le) (rate(agent_spend_USD_bucket[1h])))
What does it cost?Tokens per minutetimeseriessum by (gen_ai_token_type) (rate(gen_ai_client_token_usage_sum[5m])) * 60
What does it cost?Tool calls per turntimeseriesagent:tool_calls_per_turn:1d