Agent Harness Tutorial · End to end · Zoom level 3
Alert rules
An alert is a saved question that runs on a schedule and tells a person when the answer is "yes". On Azure today there are six, written in KQL against the Log Analytics workspace. The reference agent has a fuller set written for Prometheus. Both lists below are read from the real files.
On Azure: 6 scheduled log alerts (infra/modules/alerts.bicep)
Each runs every 15 minutes and emails the owner through the action group ag-fnol-owner. Plus the $10 monthly budget emails at 50%, 80% and a 100% forecast.
| Alert | Severity | What it means and what to do | The KQL |
|---|---|---|---|
Claims agent: a turn failedturn-failed | Sev 1 | A policyholder got "I am having trouble answering right now". Open the trace by its SessionId in App Insights > Transaction search. | AppDependencies | where Name == "agent.turn" and tostring(Properties["agent.turn.result"]) == "failed" | project TimeGenerated, conversation = SessionId, run = tostring(Properties["agent.run.id"]) |
Claims agent: a container crashed or failed to startagent-not-starting | Sev 1 | A container exited with code 1 or printed a Python traceback. Container Apps > Revisions and replicas, and the console log. | union (ContainerAppSystemLogs_CL | where Reason_s == "ContainerTerminated" and Log_s has "exit code '1'" | project TimeGenerated, app = ContainerAppName_s, revision = RevisionName_s, what = Log_s), (ContainerAppConsoleLogs_CL | where Log_s startswith "Traceback" or Log_s has "Error:" | project TimeGenerated, app = ContainerAppName_s, revision = RevisionName_s, what = Log_s) |
Claims agent: model calls failingmodel-calls-failing | Sev 2 | APIM answered a chat-completions call with an error, or the agent's model call failed. A 403 can be a Content Safety block; a 429 the rate limit; a 5xx Groq. | union
(AppRequests | where Name has "chat/completions" and not(ResultCode startswith "2")
| project TimeGenerated, what = strcat("APIM returned ", ResultCode)),
(AppDependencies | where Name == "gen_ai.chat" and Success == false
| project TimeGenerated, what = "the agent's model call failed") |
Claims agent: Content Safety blocked a call, or is unavailablecontent-safety | Sev 2 | block:<category> means a prompt or completion was refused at the gateway; unavailable means it was not screened (F0 quota spent, or the service down). | AppDependencies | where Name == "gen_ai.chat" | extend verdict = tostring(Properties["agent.gateway.x-content-safety"]) | where verdict has "block" or verdict has "unavailable" | project TimeGenerated, verdict |
Claims agent: a policyholder is waiting for a personperson-needed | Sev 3 | A conversation was handed to a person, or a payout above the automatic limit is waiting. Open /ops/desk. | union
(AppDependencies | where Name == "agent.escalation.raise"
| project TimeGenerated, what = strcat("hand-off ", tostring(Properties["agent.escalation.id"]), " (", tostring(Properties["agent.escalation.rule_id"]), ")")),
(AppDependencies | where Name == "agent.turn" and tostring(Properties["agent.turn.result"]) == "needs_approval"
| project TimeGenerated, what = strcat("payout waiting, conversation ", SessionId)) |
Claims agent: an inline check blocked a reply or a callcheck-blocked | Sev 3 | A rule refused something: injection phrasing, personal data, a promise of cover, foreign currency, a write the policyholder did not ask for. Read it: an attack, or a rule misfiring. | AppDependencies | where Name == "agent.policy" and isnotempty(tostring(Properties["agent.policy.blocked_by"])) | project TimeGenerated, rule = tostring(Properties["agent.policy.blocked_by"]), position = tostring(Properties["agent.policy.position"]) |
In the reference agent: 22 Prometheus alerts (deploy/prometheus/rules/agent.yml)
Each has a unit test run with promtool in CI. Managed Prometheus on Azure is not built yet.
| Group | Alert | Severity | For | Summary | Expression |
|---|---|---|---|---|---|
| agent-reachable | CanaryFailing | page | — | The canary's '{{ $labels.case }}' case failed twice running | sum by (case) (increase(agent_canary_cases_total{outcome="fail"}[25m])) >= 2 |
| agent-reachable | CanarySilent | page | — | The canary has not reported for 25 minutes | absent_over_time(agent_canary_cases_total[25m]) |
| agent-errors | TurnsFailing | page | — | {{ $value | humanizePercentage }} of turns failed in 10 minutes | sum(rate(agent_turns_total{synthetic="false",result="failed"}[10m])) / sum(rate(agent_turns_total{synthetic="false"}[10m])) > 0.05 and sum(increase(agent_turns_total{synthetic="false"}[10m])) >= 10 labels: { severity: page, pattern: AACP-0049 } annotations: summary: "{{ $value | humanizePercentage }} of turns failed in 10 minutes" |
| agent-errors | TurnLatencyTail | page | 10m | p95 turn latency {{ $value | humanizeDuration }} | histogram_quantile(0.95, sum by (le) (rate(agent_turn_duration_seconds_bucket{synthetic="false"}[10m]))) > 20 and sum(increase(agent_turn_duration_seconds_count{synthetic="false"}[10m])) >= 10 for: 10m labels: { severity: page, pattern: AACP-0051 } annotations: summary: "p95 turn latency {{ $value | humanizeDuration }}" |
| agent-errors | ProviderDegraded | page | — | {{ $value }} model calls found the provider unavailable in 5 minutes | sum(increase(agent_model_calls_total{outcome="unavailable"}[5m])) >= 3 |
| agent-errors | ToolRetryStorm | page | — | {{ $labels.tool }} is failing {{ $value | humanizePercentage }} of calls | (sum by (tool) (rate(agent_tool_calls_total{outcome="error"}[10m])) / sum by (tool) (rate(agent_tool_calls_total[10m]))) > 0.10 and sum by (tool) (increase(agent_tool_calls_total[10m])) >= 10 labels: { severity: page, pattern: AACP-0020 } annotations: summary: "{{ $labels.tool }} is failing {{ $value | humanizePercentage }} of calls" |
| agent-errors | ToolLatencyDrift | ticket | 15m | {{ $labels.tool }} p95 is over twice last week's | histogram_quantile(0.95, sum by (tool, le) (rate(agent_tool_duration_seconds_bucket[15m]))) > 2 * histogram_quantile(0.95, sum by (tool, le) (rate(agent_tool_duration_seconds_bucket[15m] offset 1w))) for: 15m labels: { severity: ticket, pattern: AACP-0027 } annotations: summary: "{{ $labels.tool }} p95 is over twice last week's" |
| agent-behaviour | BehaviourShift | page | — | {{ $value | humanizePercentage }} of turns ended escalated, refused or failed | (sum(rate(agent_turns_total{synthetic="false",result=~"escalated|refused|failed"}[1h])) / sum(rate(agent_turns_total{synthetic="false"}[1h]))) > 2 * (sum(rate(agent_turns_total{synthetic="false",result=~"escalated|refused|failed"}[1d] offset 1w)) / sum(rate(agent_turns_total{synthetic="false"}[1d] offset 1w))) and (sum(rate(agent_turns_total{synthetic="false",result=~"escalated|refused|failed"}[1h])) / sum(rate(agent_turns_total{synthetic="false"}[1h]))) > 0.15 and sum(increase(agent_turns_total{synthetic="false"}[1h])) >= 20 labels: { severity: page, pattern: AACP-0049 } annotations: summary: "{{ $value | humanizePercentage }} of turns ended escalated, refused or failed" description: >- Twice last week's share. Every request can succeed while the agent stops doing its job; this is that, seen as a rate. |
| agent-behaviour | RuleRefusingTooMuch | page | — | Rule {{ $labels.rule }} refused {{ $value | humanizePercentage }} of turns | sum by (rule) (rate(agent_refusals_total{synthetic="false"}[1h])) / scalar(sum(rate(agent_turns_total{synthetic="false"}[1h]))) > 0.10 and sum by (rule) (increase(agent_refusals_total{synthetic="false"}[1h])) >= 5 labels: { severity: page, pattern: AACP-0046 } annotations: summary: "Rule {{ $labels.rule }} refused {{ $value | humanizePercentage }} of turns" |
| agent-behaviour | RunsEndingOnLimits | ticket | — | {{ $value | humanizePercentage }} of runs ended on a limit, not an answer | sum(rate(agent_run_terminations_total{reason=~"step_budget_exhausted|oscillation_detected|cost_ceiling_reached"}[1d])) / sum(rate(agent_run_terminations_total[1d])) > 0.05 and sum(increase(agent_run_terminations_total[1d])) >= 50 labels: { severity: ticket, pattern: AACP-0036 } annotations: summary: "{{ $value | humanizePercentage }} of runs ended on a limit, not an answer" |
| agent-behaviour | ModelOutputUnreadable | ticket | — | {{ $value | humanizePercentage }} of model responses could not be read | sum(rate(agent_model_calls_total{outcome="malformed"}[1h])) / sum(rate(agent_model_calls_total[1h])) > 0.02 and sum(increase(agent_model_calls_total[1h])) >= 20 labels: { severity: ticket, pattern: AACP-0038 } annotations: summary: "{{ $value | humanizePercentage }} of model responses could not be read" |
| agent-behaviour | CallsPerTurnAmplified | ticket | — | Model calls per turn are half again last week's | (sum(rate(agent_model_calls_total[1d])) / sum(rate(agent_turns_total{synthetic="false"}[1d]))) > 1.5 * (sum(rate(agent_model_calls_total[1d] offset 1w)) / sum(rate(agent_turns_total{synthetic="false"}[1d] offset 1w))) labels: { severity: ticket, pattern: AACP-0001 } annotations: summary: Model calls per turn are half again last week's |
| agent-behaviour | WritesRefusedForAuthority | page | — | {{ $labels.tool }} was refused as an error, not a no | sum by (tool) (increase(agent_tool_calls_total{tool=~"cancel_order|issue_refund|open_return_request|change_address",outcome="error"}[15m])) > 0 labels: { severity: page, pattern: AACP-0047 } annotations: summary: "{{ $labels.tool }} was refused as an error, not a no" description: >- A write the far end rejected outright — no approval, no scope, or not reachable — rather than answering allowed: false. The far end's check is the only thing that held. |
| agent-behaviour | EscalationRuleNotNeeded | ticket | — | The desk said the agent could have handled {{ $value | humanizePercentage }} of rule {{ $labels.rule }}'s hand-offs | sum by (rule) (increase(agent_escalations_closed_total{outcome="agent_could_have"}[1w])) / sum by (rule) (increase(agent_escalations_closed_total{state="resolved"}[1w])) > 0.30 and sum by (rule) (increase(agent_escalations_closed_total{state="resolved"}[1w])) >= 10 labels: { severity: ticket, pattern: AACP-0044 } annotations: summary: "The desk said the agent could have handled {{ $value | humanizePercentage }} of rule {{ $labels.rule }}'s hand-offs" description: >- One rule's escalations are mostly ones a person closed as unneeded. The rule is written too broad, and its cost falls on the desk. |
| agent-behaviour | RatesMovedAfterRelease | page | — | Configuration {{ $labels.config }}: {{ $value | humanizePercentage }} of turns ended escalated, refused or failed | (sum by (config) (rate(agent_turns_total{synthetic="false",result=~"escalated|refused|failed"}[1h])) / sum by (config) (rate(agent_turns_total{synthetic="false"}[1h]))) > 2 * scalar(sum(rate(agent_turns_total{synthetic="false",result=~"escalated|refused|failed"}[1d] offset 1d)) / sum(rate(agent_turns_total{synthetic="false"}[1d] offset 1d))) and (sum by (config) (rate(agent_turns_total{synthetic="false",result=~"escalated|refused|failed"}[1h])) / sum by (config) (rate(agent_turns_total{synthetic="false"}[1h]))) > 0.15 and sum by (config) (increase(agent_turns_total{synthetic="false"}[1h])) >= 20 unless on (config) sum by (config) (increase(agent_turns_total{synthetic="false"}[1d] offset 1d)) > 0 labels: { severity: page, pattern: AACP-0055 } annotations: summary: "Configuration {{ $labels.config }}: {{ $value | humanizePercentage }} of turns ended escalated, refused or failed" description: >- A configuration that is less than a day old moved the rates to twice yesterday's. The release is the first suspect. Spend is not yet split by configuration, so only the turn rates are compared here. |
| agent-behaviour | WaitsLapsing | ticket | — | {{ $value }} approvals or escalations expired with nobody in 6 hours | (sum(increase(agent_approvals_settled_total{outcome="expired"}[6h])) or vector(0)) + (sum(increase(agent_escalations_closed_total{state="expired"}[6h])) or vector(0)) > 0 labels: { severity: ticket, pattern: AACP-0045 } annotations: summary: "{{ $value }} approvals or escalations expired with nobody in 6 hours" |
| agent-tool-surface | UnofferedToolsRequested | ticket | — | {{ $value | humanizePercentage }} of tool calls asked for a tool that is not offered | sum(rate(agent_tool_calls_total{tool="unoffered"}[1h])) / sum(rate(agent_tool_calls_total[1h])) > 0.01 and sum(increase(agent_tool_calls_total[1h])) >= 50 labels: { severity: ticket, pattern: AACP-0023 } annotations: summary: "{{ $value | humanizePercentage }} of tool calls asked for a tool that is not offered" description: >- The model is asking for tools by names it has seen elsewhere, most often after a model or prompt change. |
| agent-tool-surface | ToolArgumentsFailingSchema | ticket | — | {{ $labels.tool }}: {{ $value | humanizePercentage }} of calls failed the tool's own schema | sum by (tool) (rate(agent_tool_invalid_arguments_total[1h])) / sum by (tool) (rate(agent_tool_calls_total[1h])) > 0.05 and sum by (tool) (increase(agent_tool_calls_total[1h])) >= 20 labels: { severity: ticket, pattern: AACP-0024 } annotations: summary: "{{ $labels.tool }}: {{ $value | humanizePercentage }} of calls failed the tool's own schema" |
| agent-telemetry | TelemetryDropping | ticket | — | {{ $value }} spans could not be delivered to the trace store in 15 minutes | sum(increase({__name__=~"otelcol_exporter_(send|enqueue)_failed_spans.*"}[15m])) > 0 labels: { severity: ticket, pattern: AACP-0056 } annotations: summary: "{{ $value }} spans could not be delivered to the trace store in 15 minutes" description: >- Those turns are missing from the trace store, so the online rules never saw them and every rate computed there is short. |
| agent-cost | SpendPerRunDoubled | ticket | — | p95 cost per run is {{ $value }} USD, twice last week's | histogram_quantile(0.95, sum by (le) (rate(agent_spend_USD_bucket[1h]))) > 2 * histogram_quantile(0.95, sum by (le) (rate(agent_spend_USD_bucket[1d] offset 1w))) and sum(increase(agent_spend_USD_count[1h])) >= 20 labels: { severity: ticket, pattern: AACP-0053 } annotations: summary: "p95 cost per run is {{ $value }} USD, twice last week's" |
| agent-online | OnlineRulePage | page | — | Online rule {{ $labels.rule }} found a turn a customer was misled in | sum by (rule) (increase(agent_online_findings_total{severity="page"}[15m])) > 0 |
| agent-online | OnlineRuleTickets | ticket | — | Online rule {{ $labels.rule }} fired {{ $value }} times today | sum by (rule) (increase(agent_online_findings_total{severity="ticket"}[1d])) >= 5 |
Recording rules (pre-computed series the dashboard reads)
| Series | Expression |
|---|---|
agent:returned_share:1d | sum(increase(agent_outcomes_total{kind="returned"}[1d])) / clamp_min(sum(increase(agent_online_evaluated_total[1d])), 1) labels: { pattern: AACP-0041 } |
agent:asked_for_person_share:1d | sum(increase(agent_outcomes_total{kind="asked_for_person"}[1d])) / clamp_min(sum(increase(agent_online_evaluated_total[1d])), 1) labels: { pattern: AACP-0042 } |
agent:tool_calls_per_turn:1d | sum(rate(agent_tool_calls_total[1d])) / clamp_min(sum(rate(agent_turns_total{synthetic="false"}[1d])), 1e-9) labels: { pattern: AACP-0037 } |
agent:feedback_down_share:1d | sum(increase(agent_outcomes_total{kind="feedback_down"}[1d])) / clamp_min(sum(increase(agent_outcomes_total{kind=~"feedback_.*"}[1d])), 1) labels: { pattern: AACP-0041 } |
agent:input_tokens_median_vs_last_week:1d | histogram_quantile(0.5, sum by (config, le) (rate(gen_ai_client_token_usage_bucket{gen_ai_token_type="input"}[1d]))) / scalar(histogram_quantile(0.5, sum by (le) (rate(gen_ai_client_token_usage_bucket{gen_ai_token_type="input"}[1d] offset 1w)))) labels: { pattern: AACP-0002 } |
agent:tool_truncated_share:1d | sum by (tool) (increase(agent_tool_truncated_total[1d])) / clamp_min(sum by (tool) (increase(agent_tool_calls_total[1d])), 1) labels: { pattern: AACP-0004 } |
agent:tool_offered_unused:7d | (sum by (tool) (increase(agent_tools_offered_total[7d])) > 0) unless on (tool) (sum by (tool) (increase(agent_tool_calls_total{outcome="ok"}[7d])) > 0) and on () (sum(increase(agent_turns_total{synthetic="false"}[7d])) >= 500) labels: { pattern: AACP-0026 } |