Agent Harness Tutorial · End to end · Zoom level 3
Metrics: counters and histograms
Spans answer "what happened in this one turn?". Metrics answer "how often, how fast, how much, across all turns?". Every metric is a counter (it only goes up) or a histogram (it records a spread of values, like durations or cost). Labels are kept to a few known values, such as the route, the rule or the outcome; ids never go on a metric.
| Metric | Kind | Unit | What it counts | Name in Prometheus |
|---|---|---|---|---|
agent.turns | counter | — | Turns served, by how each one ended. | agent_turns_total |
agent.turn.duration | histogram | s | How long a turn took, from the customer's words to the reply. | agent_turn_duration |
agent.escalations | counter | — | Conversations handed to a person, by the rule that decided. | agent_escalations_total |
agent.refusals | counter | — | Requests refused, by the rule that refused them. | agent_refusals_total |
agent.approvals | counter | — | Approval decisions, by what was decided. | agent_approvals_total |
agent.unbacked_promises | counter | — | agent_unbacked_promises_total | |
agent.spend | histogram | USD | What one run spent with the provider. | agent_spend |
agent.model.calls | counter | — | Calls to the provider, and how many came back unreadable. | agent_model_calls_total |
gen_ai.client.operation.duration | histogram | s | GenAI semantic conventions: duration of a model call. | gen_ai_client_operation_duration |
gen_ai.client.token.usage | histogram | {token} | GenAI semantic conventions: tokens per model call, by type. | gen_ai_client_token_usage |
agent.tool.calls | counter | — | Tool calls, by tool and by what came back. | agent_tool_calls_total |
agent.tool.duration | histogram | s | How long a tool call took, measured at the agent. | agent_tool_duration |
agent.run.terminations | counter | — | Why each run stopped. | agent_run_terminations_total |
agent.online.findings | counter | — | What the online rules found in evaluated turns, by rule. | agent_online_findings_total |
agent.online.evaluated | counter | — | Turns the online rules evaluated. | agent_online_evaluated_total |
agent.outcomes | counter | — | agent_outcomes_total | |
agent.canary.cases | counter | — | agent_canary_cases_total | |
agent.tools.offered | counter | — | agent_tools_offered_total | |
agent.tool.truncated | counter | — | agent_tool_truncated_total | |
agent.tool.invalid_arguments | counter | — | agent_tool_invalid_arguments_total |
On Azure they go through the OpenTelemetry distro into Application Insights, stored in the AppMetrics table of the Log Analytics workspace (see where it lands). Histogram bucket edges are dropped there, so a p95 on Azure is computed from span durations instead. APIM adds its own token metric (llm-emit-token-metric, namespace claims-fnol).
In the reference agent they are scraped by Prometheus every 15 seconds, which is what the alert rules and dashboard read. Managed Prometheus on Azure is not built yet (Tier 12).