cleandataengineering.aiAgent Harness, drawn simply
You are here: End to end › Eval › Online › Alert rules

Agent Harness Tutorial · End to end · Zoom level 3

Alert rules

An alert is a saved question that runs on a schedule and tells a person when the answer is "yes". On Azure today there are six, written in KQL against the Log Analytics workspace. The reference agent has a fuller set written for Prometheus. Both lists below are read from the real files.

On Azure: 6 scheduled log alerts (infra/modules/alerts.bicep)

Each runs every 15 minutes and emails the owner through the action group ag-fnol-owner. Plus the $10 monthly budget emails at 50%, 80% and a 100% forecast.

AlertSeverityWhat it means and what to doThe KQL
Claims agent: a turn failed
turn-failed
Sev 1A policyholder got "I am having trouble answering right now". Open the trace by its SessionId in App Insights > Transaction search.
AppDependencies
| where Name == "agent.turn" and tostring(Properties["agent.turn.result"]) == "failed"
| project TimeGenerated, conversation = SessionId, run = tostring(Properties["agent.run.id"])
Claims agent: a container crashed or failed to start
agent-not-starting
Sev 1A container exited with code 1 or printed a Python traceback. Container Apps > Revisions and replicas, and the console log.
union
  (ContainerAppSystemLogs_CL | where Reason_s == "ContainerTerminated" and Log_s has "exit code '1'"
   | project TimeGenerated, app = ContainerAppName_s, revision = RevisionName_s, what = Log_s),
  (ContainerAppConsoleLogs_CL | where Log_s startswith "Traceback" or Log_s has "Error:"
   | project TimeGenerated, app = ContainerAppName_s, revision = RevisionName_s, what = Log_s)
Claims agent: model calls failing
model-calls-failing
Sev 2APIM answered a chat-completions call with an error, or the agent's model call failed. A 403 can be a Content Safety block; a 429 the rate limit; a 5xx Groq.
union
  (AppRequests | where Name has "chat/completions" and not(ResultCode startswith "2")
   | project TimeGenerated, what = strcat("APIM returned ", ResultCode)),
  (AppDependencies | where Name == "gen_ai.chat" and Success == false
   | project TimeGenerated, what = "the agent's model call failed")
Claims agent: Content Safety blocked a call, or is unavailable
content-safety
Sev 2block:<category> means a prompt or completion was refused at the gateway; unavailable means it was not screened (F0 quota spent, or the service down).
AppDependencies
| where Name == "gen_ai.chat"
| extend verdict = tostring(Properties["agent.gateway.x-content-safety"])
| where verdict has "block" or verdict has "unavailable"
| project TimeGenerated, verdict
Claims agent: a policyholder is waiting for a person
person-needed
Sev 3A conversation was handed to a person, or a payout above the automatic limit is waiting. Open /ops/desk.
union
  (AppDependencies | where Name == "agent.escalation.raise"
   | project TimeGenerated, what = strcat("hand-off ", tostring(Properties["agent.escalation.id"]), " (", tostring(Properties["agent.escalation.rule_id"]), ")")),
  (AppDependencies | where Name == "agent.turn" and tostring(Properties["agent.turn.result"]) == "needs_approval"
   | project TimeGenerated, what = strcat("payout waiting, conversation ", SessionId))
Claims agent: an inline check blocked a reply or a call
check-blocked
Sev 3A rule refused something: injection phrasing, personal data, a promise of cover, foreign currency, a write the policyholder did not ask for. Read it: an attack, or a rule misfiring.
AppDependencies
| where Name == "agent.policy" and isnotempty(tostring(Properties["agent.policy.blocked_by"]))
| project TimeGenerated, rule = tostring(Properties["agent.policy.blocked_by"]), position = tostring(Properties["agent.policy.position"])

In the reference agent: 22 Prometheus alerts (deploy/prometheus/rules/agent.yml)

Each has a unit test run with promtool in CI. Managed Prometheus on Azure is not built yet.

GroupAlertSeverityForSummaryExpression
agent-reachableCanaryFailingpage—The canary's '{{ $labels.case }}' case failed twice runningsum by (case) (increase(agent_canary_cases_total{outcome="fail"}[25m])) >= 2
agent-reachableCanarySilentpage—The canary has not reported for 25 minutesabsent_over_time(agent_canary_cases_total[25m])
agent-errorsTurnsFailingpage—{{ $value | humanizePercentage }} of turns failed in 10 minutessum(rate(agent_turns_total{synthetic="false",result="failed"}[10m])) / sum(rate(agent_turns_total{synthetic="false"}[10m])) > 0.05 and sum(increase(agent_turns_total{synthetic="false"}[10m])) >= 10 labels: { severity: page, pattern: AACP-0049 } annotations: summary: "{{ $value | humanizePercentage }} of turns failed in 10 minutes"
agent-errorsTurnLatencyTailpage10mp95 turn latency {{ $value | humanizeDuration }}histogram_quantile(0.95, sum by (le) (rate(agent_turn_duration_seconds_bucket{synthetic="false"}[10m]))) > 20 and sum(increase(agent_turn_duration_seconds_count{synthetic="false"}[10m])) >= 10 for: 10m labels: { severity: page, pattern: AACP-0051 } annotations: summary: "p95 turn latency {{ $value | humanizeDuration }}"
agent-errorsProviderDegradedpage—{{ $value }} model calls found the provider unavailable in 5 minutessum(increase(agent_model_calls_total{outcome="unavailable"}[5m])) >= 3
agent-errorsToolRetryStormpage—{{ $labels.tool }} is failing {{ $value | humanizePercentage }} of calls(sum by (tool) (rate(agent_tool_calls_total{outcome="error"}[10m])) / sum by (tool) (rate(agent_tool_calls_total[10m]))) > 0.10 and sum by (tool) (increase(agent_tool_calls_total[10m])) >= 10 labels: { severity: page, pattern: AACP-0020 } annotations: summary: "{{ $labels.tool }} is failing {{ $value | humanizePercentage }} of calls"
agent-errorsToolLatencyDriftticket15m{{ $labels.tool }} p95 is over twice last week'shistogram_quantile(0.95, sum by (tool, le) (rate(agent_tool_duration_seconds_bucket[15m]))) > 2 * histogram_quantile(0.95, sum by (tool, le) (rate(agent_tool_duration_seconds_bucket[15m] offset 1w))) for: 15m labels: { severity: ticket, pattern: AACP-0027 } annotations: summary: "{{ $labels.tool }} p95 is over twice last week's"
agent-behaviourBehaviourShiftpage—{{ $value | humanizePercentage }} of turns ended escalated, refused or failed(sum(rate(agent_turns_total{synthetic="false",result=~"escalated|refused|failed"}[1h])) / sum(rate(agent_turns_total{synthetic="false"}[1h]))) > 2 * (sum(rate(agent_turns_total{synthetic="false",result=~"escalated|refused|failed"}[1d] offset 1w)) / sum(rate(agent_turns_total{synthetic="false"}[1d] offset 1w))) and (sum(rate(agent_turns_total{synthetic="false",result=~"escalated|refused|failed"}[1h])) / sum(rate(agent_turns_total{synthetic="false"}[1h]))) > 0.15 and sum(increase(agent_turns_total{synthetic="false"}[1h])) >= 20 labels: { severity: page, pattern: AACP-0049 } annotations: summary: "{{ $value | humanizePercentage }} of turns ended escalated, refused or failed" description: >- Twice last week's share. Every request can succeed while the agent stops doing its job; this is that, seen as a rate.
agent-behaviourRuleRefusingTooMuchpage—Rule {{ $labels.rule }} refused {{ $value | humanizePercentage }} of turnssum by (rule) (rate(agent_refusals_total{synthetic="false"}[1h])) / scalar(sum(rate(agent_turns_total{synthetic="false"}[1h]))) > 0.10 and sum by (rule) (increase(agent_refusals_total{synthetic="false"}[1h])) >= 5 labels: { severity: page, pattern: AACP-0046 } annotations: summary: "Rule {{ $labels.rule }} refused {{ $value | humanizePercentage }} of turns"
agent-behaviourRunsEndingOnLimitsticket—{{ $value | humanizePercentage }} of runs ended on a limit, not an answersum(rate(agent_run_terminations_total{reason=~"step_budget_exhausted|oscillation_detected|cost_ceiling_reached"}[1d])) / sum(rate(agent_run_terminations_total[1d])) > 0.05 and sum(increase(agent_run_terminations_total[1d])) >= 50 labels: { severity: ticket, pattern: AACP-0036 } annotations: summary: "{{ $value | humanizePercentage }} of runs ended on a limit, not an answer"
agent-behaviourModelOutputUnreadableticket—{{ $value | humanizePercentage }} of model responses could not be readsum(rate(agent_model_calls_total{outcome="malformed"}[1h])) / sum(rate(agent_model_calls_total[1h])) > 0.02 and sum(increase(agent_model_calls_total[1h])) >= 20 labels: { severity: ticket, pattern: AACP-0038 } annotations: summary: "{{ $value | humanizePercentage }} of model responses could not be read"
agent-behaviourCallsPerTurnAmplifiedticket—Model calls per turn are half again last week's(sum(rate(agent_model_calls_total[1d])) / sum(rate(agent_turns_total{synthetic="false"}[1d]))) > 1.5 * (sum(rate(agent_model_calls_total[1d] offset 1w)) / sum(rate(agent_turns_total{synthetic="false"}[1d] offset 1w))) labels: { severity: ticket, pattern: AACP-0001 } annotations: summary: Model calls per turn are half again last week's
agent-behaviourWritesRefusedForAuthoritypage—{{ $labels.tool }} was refused as an error, not a nosum by (tool) (increase(agent_tool_calls_total{tool=~"cancel_order|issue_refund|open_return_request|change_address",outcome="error"}[15m])) > 0 labels: { severity: page, pattern: AACP-0047 } annotations: summary: "{{ $labels.tool }} was refused as an error, not a no" description: >- A write the far end rejected outright — no approval, no scope, or not reachable — rather than answering allowed: false. The far end's check is the only thing that held.
agent-behaviourEscalationRuleNotNeededticket—The desk said the agent could have handled {{ $value | humanizePercentage }} of rule {{ $labels.rule }}'s hand-offssum by (rule) (increase(agent_escalations_closed_total{outcome="agent_could_have"}[1w])) / sum by (rule) (increase(agent_escalations_closed_total{state="resolved"}[1w])) > 0.30 and sum by (rule) (increase(agent_escalations_closed_total{state="resolved"}[1w])) >= 10 labels: { severity: ticket, pattern: AACP-0044 } annotations: summary: "The desk said the agent could have handled {{ $value | humanizePercentage }} of rule {{ $labels.rule }}'s hand-offs" description: >- One rule's escalations are mostly ones a person closed as unneeded. The rule is written too broad, and its cost falls on the desk.
agent-behaviourRatesMovedAfterReleasepage—Configuration {{ $labels.config }}: {{ $value | humanizePercentage }} of turns ended escalated, refused or failed(sum by (config) (rate(agent_turns_total{synthetic="false",result=~"escalated|refused|failed"}[1h])) / sum by (config) (rate(agent_turns_total{synthetic="false"}[1h]))) > 2 * scalar(sum(rate(agent_turns_total{synthetic="false",result=~"escalated|refused|failed"}[1d] offset 1d)) / sum(rate(agent_turns_total{synthetic="false"}[1d] offset 1d))) and (sum by (config) (rate(agent_turns_total{synthetic="false",result=~"escalated|refused|failed"}[1h])) / sum by (config) (rate(agent_turns_total{synthetic="false"}[1h]))) > 0.15 and sum by (config) (increase(agent_turns_total{synthetic="false"}[1h])) >= 20 unless on (config) sum by (config) (increase(agent_turns_total{synthetic="false"}[1d] offset 1d)) > 0 labels: { severity: page, pattern: AACP-0055 } annotations: summary: "Configuration {{ $labels.config }}: {{ $value | humanizePercentage }} of turns ended escalated, refused or failed" description: >- A configuration that is less than a day old moved the rates to twice yesterday's. The release is the first suspect. Spend is not yet split by configuration, so only the turn rates are compared here.
agent-behaviourWaitsLapsingticket—{{ $value }} approvals or escalations expired with nobody in 6 hours(sum(increase(agent_approvals_settled_total{outcome="expired"}[6h])) or vector(0)) + (sum(increase(agent_escalations_closed_total{state="expired"}[6h])) or vector(0)) > 0 labels: { severity: ticket, pattern: AACP-0045 } annotations: summary: "{{ $value }} approvals or escalations expired with nobody in 6 hours"
agent-tool-surfaceUnofferedToolsRequestedticket—{{ $value | humanizePercentage }} of tool calls asked for a tool that is not offeredsum(rate(agent_tool_calls_total{tool="unoffered"}[1h])) / sum(rate(agent_tool_calls_total[1h])) > 0.01 and sum(increase(agent_tool_calls_total[1h])) >= 50 labels: { severity: ticket, pattern: AACP-0023 } annotations: summary: "{{ $value | humanizePercentage }} of tool calls asked for a tool that is not offered" description: >- The model is asking for tools by names it has seen elsewhere, most often after a model or prompt change.
agent-tool-surfaceToolArgumentsFailingSchematicket—{{ $labels.tool }}: {{ $value | humanizePercentage }} of calls failed the tool's own schemasum by (tool) (rate(agent_tool_invalid_arguments_total[1h])) / sum by (tool) (rate(agent_tool_calls_total[1h])) > 0.05 and sum by (tool) (increase(agent_tool_calls_total[1h])) >= 20 labels: { severity: ticket, pattern: AACP-0024 } annotations: summary: "{{ $labels.tool }}: {{ $value | humanizePercentage }} of calls failed the tool's own schema"
agent-telemetryTelemetryDroppingticket—{{ $value }} spans could not be delivered to the trace store in 15 minutessum(increase({__name__=~"otelcol_exporter_(send|enqueue)_failed_spans.*"}[15m])) > 0 labels: { severity: ticket, pattern: AACP-0056 } annotations: summary: "{{ $value }} spans could not be delivered to the trace store in 15 minutes" description: >- Those turns are missing from the trace store, so the online rules never saw them and every rate computed there is short.
agent-costSpendPerRunDoubledticket—p95 cost per run is {{ $value }} USD, twice last week'shistogram_quantile(0.95, sum by (le) (rate(agent_spend_USD_bucket[1h]))) > 2 * histogram_quantile(0.95, sum by (le) (rate(agent_spend_USD_bucket[1d] offset 1w))) and sum(increase(agent_spend_USD_count[1h])) >= 20 labels: { severity: ticket, pattern: AACP-0053 } annotations: summary: "p95 cost per run is {{ $value }} USD, twice last week's"
agent-onlineOnlineRulePagepage—Online rule {{ $labels.rule }} found a turn a customer was misled insum by (rule) (increase(agent_online_findings_total{severity="page"}[15m])) > 0
agent-onlineOnlineRuleTicketsticket—Online rule {{ $labels.rule }} fired {{ $value }} times todaysum by (rule) (increase(agent_online_findings_total{severity="ticket"}[1d])) >= 5

Recording rules (pre-computed series the dashboard reads)

SeriesExpression
agent:returned_share:1dsum(increase(agent_outcomes_total{kind="returned"}[1d])) / clamp_min(sum(increase(agent_online_evaluated_total[1d])), 1) labels: { pattern: AACP-0041 }
agent:asked_for_person_share:1dsum(increase(agent_outcomes_total{kind="asked_for_person"}[1d])) / clamp_min(sum(increase(agent_online_evaluated_total[1d])), 1) labels: { pattern: AACP-0042 }
agent:tool_calls_per_turn:1dsum(rate(agent_tool_calls_total[1d])) / clamp_min(sum(rate(agent_turns_total{synthetic="false"}[1d])), 1e-9) labels: { pattern: AACP-0037 }
agent:feedback_down_share:1dsum(increase(agent_outcomes_total{kind="feedback_down"}[1d])) / clamp_min(sum(increase(agent_outcomes_total{kind=~"feedback_.*"}[1d])), 1) labels: { pattern: AACP-0041 }
agent:input_tokens_median_vs_last_week:1dhistogram_quantile(0.5, sum by (config, le) (rate(gen_ai_client_token_usage_bucket{gen_ai_token_type="input"}[1d]))) / scalar(histogram_quantile(0.5, sum by (le) (rate(gen_ai_client_token_usage_bucket{gen_ai_token_type="input"}[1d] offset 1w)))) labels: { pattern: AACP-0002 }
agent:tool_truncated_share:1dsum by (tool) (increase(agent_tool_truncated_total[1d])) / clamp_min(sum by (tool) (increase(agent_tool_calls_total[1d])), 1) labels: { pattern: AACP-0004 }
agent:tool_offered_unused:7d(sum by (tool) (increase(agent_tools_offered_total[7d])) > 0) unless on (tool) (sum by (tool) (increase(agent_tool_calls_total{outcome="ok"}[7d])) > 0) and on () (sum(increase(agent_turns_total{synthetic="false"}[7d])) >= 500) labels: { pattern: AACP-0026 }