cleandataengineering.aiAgent Harness, drawn simply
You are here: End to end › Eval › Online

Agent Harness Tutorial · End to end · Zoom level 2

Online eval

Some mistakes can only be seen after the turn, or across many turns: a slow drift in cost, a rule that suddenly refuses too much, a reply that contradicts the claims system. Online eval re-reads finished turns and raises an alert.

Finished turnsin App Insightsevery 60 sThe watch jobrebuild each turnrun 24 rulesFindingspage · ticket· trendAlertemail, dashboard
It runs beside the agent and can never change a reply already sent.
SeverityExample rules
Page someonethe reply contradicts the claims system · claims an action that didn't happen · personal data in a reply · a write after instructions arrived in a result
Open a ticketended by a limit · answered over a failed tool · the same failing tool called again · an empty answer · an unbacked promise
Watch the trendslow turns · costly turns · Rohan repeating himself · later turns costing more

Status, honestly. The watch job with its 24 rules runs in the reference agent locally. On Azure today, only the budget alert emails; the watch job, a findings table and Azure Monitor alerts are not built yet (planned). A sampled LLM judge in Azure AI Foundry is planned later.

How it is done: the real code

Cut from the live repositories each time the site is built, so it always matches what runs.

claims_fnol/policy/evaluators.yaml:55What re-runs online on stored turns
  online:
    sample: 1.0
    run:
      - {use: no_ungrounded_entity, on_fail: alert}
      - {use: money_in_rupees, on_fail: alert}
      - {use: content_safety, on_fail: alert}   # A9: online only, never at reply (F-79)
      - {use: no_known_injection, on_fail: alert}
      - {use: no_instructions_in_result, on_fail: alert}
infra/modules/alerts.bicep:1infra/modules/alerts.bicep: the log alerts
// WHAT IT IS
//   Email alerts on the agent's own telemetry: one action group (email to the
//   owner) and six log alert rules, each a KQL query over the Log Analytics
//   workspace App Insights writes to, run every 15 minutes. A rule fires when its
//   query returns any row in the last 15 minutes:
//     turn-failed           a turn ended Failed ("I am having trouble answering")
//     agent-not-starting    a container exited with code 1, or printed a traceback
//                           (F-118: the new revision failed and nobody knew)
//     model-calls-failing   APIM answered a model call with 4xx/5xx, or the agent's
//                           model call failed
//     content-safety        Content Safety blocked a prompt or completion, or was
//                           unavailable (the F0 quota spent: calls pass unscreened)
//     person-needed         a hand-off was raised, or a payout is waiting for a
//                           claims handler — someone has to act
//     check-blocked         an inline check blocked (injection, PII, a promise of
//                           cover, money in £/$/€…): an attack or a rule misfiring
//
// WHICH CONCERN IT SERVES
//   Observability (deck: rule → finding → alert; PLAN Tier 4b B3 "log alerts").
//   Until this, App Insights only collected: the $10 budget was the only alert.
//   The online watch rules (findings table, Workbook, canary) are still Tier 4b.
//
// EXPECTED DEV COST
//   ~$3/month: log search alert rules at 15-minute frequency are $0.50 a rule a
//   month (Azure Retail Prices API, Central India, "Alerts System Log Monitored at
//   15 Minute Frequency", read 14 Oct 2026); emails are free for the first 1,000
//   a month ("Emails"). Five-minute rules would be $1.50 each.
//
// WHERE TO LOOK (portal tour)
//   Monitor > Alerts (fired alerts) · Monitor > Alerts > Alert rules (these six) ·

Look inside