Agent Harness Tutorial · End to end · Zoom level 2
Online eval
Some mistakes can only be seen after the turn, or across many turns: a slow drift in cost, a rule that suddenly refuses too much, a reply that contradicts the claims system. Online eval re-reads finished turns and raises an alert.
| Severity | Example rules |
|---|---|
| Page someone | the reply contradicts the claims system · claims an action that didn't happen · personal data in a reply · a write after instructions arrived in a result |
| Open a ticket | ended by a limit · answered over a failed tool · the same failing tool called again · an empty answer · an unbacked promise |
| Watch the trend | slow turns · costly turns · Rohan repeating himself · later turns costing more |
Status, honestly. The watch job with its 24 rules runs in the reference agent locally. On Azure today, only the budget alert emails; the watch job, a findings table and Azure Monitor alerts are not built yet (planned). A sampled LLM judge in Azure AI Foundry is planned later.
How it is done: the real code
Cut from the live repositories each time the site is built, so it always matches what runs.
claims_fnol/policy/evaluators.yaml:55What re-runs online on stored turns online:
sample: 1.0
run:
- {use: no_ungrounded_entity, on_fail: alert}
- {use: money_in_rupees, on_fail: alert}
- {use: content_safety, on_fail: alert} # A9: online only, never at reply (F-79)
- {use: no_known_injection, on_fail: alert}
- {use: no_instructions_in_result, on_fail: alert}infra/modules/alerts.bicep:1infra/modules/alerts.bicep: the log alerts// WHAT IT IS
// Email alerts on the agent's own telemetry: one action group (email to the
// owner) and six log alert rules, each a KQL query over the Log Analytics
// workspace App Insights writes to, run every 15 minutes. A rule fires when its
// query returns any row in the last 15 minutes:
// turn-failed a turn ended Failed ("I am having trouble answering")
// agent-not-starting a container exited with code 1, or printed a traceback
// (F-118: the new revision failed and nobody knew)
// model-calls-failing APIM answered a model call with 4xx/5xx, or the agent's
// model call failed
// content-safety Content Safety blocked a prompt or completion, or was
// unavailable (the F0 quota spent: calls pass unscreened)
// person-needed a hand-off was raised, or a payout is waiting for a
// claims handler — someone has to act
// check-blocked an inline check blocked (injection, PII, a promise of
// cover, money in £/$/€…): an attack or a rule misfiring
//
// WHICH CONCERN IT SERVES
// Observability (deck: rule → finding → alert; PLAN Tier 4b B3 "log alerts").
// Until this, App Insights only collected: the $10 budget was the only alert.
// The online watch rules (findings table, Workbook, canary) are still Tier 4b.
//
// EXPECTED DEV COST
// ~$3/month: log search alert rules at 15-minute frequency are $0.50 a rule a
// month (Azure Retail Prices API, Central India, "Alerts System Log Monitored at
// 15 Minute Frequency", read 14 Oct 2026); emails are free for the first 1,000
// a month ("Emails"). Five-minute rules would be $1.50 each.
//
// WHERE TO LOOK (portal tour)
// Monitor > Alerts (fired alerts) · Monitor > Alerts > Alert rules (these six) ·