cleandataengineering.aiAgent Harness, drawn simply
You are here: End to end › Eval › The trace › Metrics

Agent Harness Tutorial · End to end · Zoom level 3

Metrics: counters and histograms

Spans answer "what happened in this one turn?". Metrics answer "how often, how fast, how much, across all turns?". Every metric is a counter (it only goes up) or a histogram (it records a spread of values, like durations or cost). Labels are kept to a few known values, such as the route, the rule or the outcome; ids never go on a metric.

MetricKindUnitWhat it countsName in Prometheus
agent.turnscounter—Turns served, by how each one ended.agent_turns_total
agent.turn.durationhistogramsHow long a turn took, from the customer's words to the reply.agent_turn_duration
agent.escalationscounter—Conversations handed to a person, by the rule that decided.agent_escalations_total
agent.refusalscounter—Requests refused, by the rule that refused them.agent_refusals_total
agent.approvalscounter—Approval decisions, by what was decided.agent_approvals_total
agent.unbacked_promisescounter—agent_unbacked_promises_total
agent.spendhistogramUSDWhat one run spent with the provider.agent_spend
agent.model.callscounter—Calls to the provider, and how many came back unreadable.agent_model_calls_total
gen_ai.client.operation.durationhistogramsGenAI semantic conventions: duration of a model call.gen_ai_client_operation_duration
gen_ai.client.token.usagehistogram{token}GenAI semantic conventions: tokens per model call, by type.gen_ai_client_token_usage
agent.tool.callscounter—Tool calls, by tool and by what came back.agent_tool_calls_total
agent.tool.durationhistogramsHow long a tool call took, measured at the agent.agent_tool_duration
agent.run.terminationscounter—Why each run stopped.agent_run_terminations_total
agent.online.findingscounter—What the online rules found in evaluated turns, by rule.agent_online_findings_total
agent.online.evaluatedcounter—Turns the online rules evaluated.agent_online_evaluated_total
agent.outcomescounter—agent_outcomes_total
agent.canary.casescounter—agent_canary_cases_total
agent.tools.offeredcounter—agent_tools_offered_total
agent.tool.truncatedcounter—agent_tool_truncated_total
agent.tool.invalid_argumentscounter—agent_tool_invalid_arguments_total

On Azure they go through the OpenTelemetry distro into Application Insights, stored in the AppMetrics table of the Log Analytics workspace (see where it lands). Histogram bucket edges are dropped there, so a p95 on Azure is computed from span durations instead. APIM adds its own token metric (llm-emit-token-metric, namespace claims-fnol).

In the reference agent they are scraped by Prometheus every 15 seconds, which is what the alert rules and dashboard read. Managed Prometheus on Azure is not built yet (Tier 12).