Agent Harness Tutorial · Cost
Who checks cost, and when
Cost is checked at six speeds, just like quality in eval: inside the turn, minutes later, at every release, daily, and monthly.
The daily engine explains a change the way finance does: a baseline (median of the last 4 same weekdays), then a split into five drivers that add up exactly (volume, model mix, size, cache and tier, price), then a drill-down by agent, model and key. Example finding: "Up 23%, because traffic moved to a pricier model on the claims key." Every number comes from the engine; a model only polishes the wording.
Alerts. In the reference agent: SpendPerRunDoubled (p95 cost per run is twice last week's, at least 20 runs in the hour) and RunsEndingOnLimits (over 5% of runs ending on a budget). On Azure today only the $10 budget emails are about money.
How it is done: the real code
Cut from the live repositories each time the site is built, so it always matches what runs.
infra/modules/alerts.bicep:1The log alerts on Azure// WHAT IT IS
// Email alerts on the agent's own telemetry: one action group (email to the
// owner) and six log alert rules, each a KQL query over the Log Analytics
// workspace App Insights writes to, run every 15 minutes. A rule fires when its
// query returns any row in the last 15 minutes:
// turn-failed a turn ended Failed ("I am having trouble answering")
// agent-not-starting a container exited with code 1, or printed a traceback
// (F-118: the new revision failed and nobody knew)
// model-calls-failing APIM answered a model call with 4xx/5xx, or the agent's
// model call failed
// content-safety Content Safety blocked a prompt or completion, or was
// unavailable (the F0 quota spent: calls pass unscreened)
// person-needed a hand-off was raised, or a payout is waiting for a
// claims handler — someone has to act
// check-blocked an inline check blocked (injection, PII, a promise of
// cover, money in £/$/€…): an attack or a rule misfiring
//
// WHICH CONCERN IT SERVES
// Observability (deck: rule → finding → alert; PLAN Tier 4b B3 "log alerts").
// Until this, App Insights only collected: the $10 budget was the only alert.
// The online watch rules (findings table, Workbook, canary) are still Tier 4b.
//
// EXPECTED DEV COST
// ~$3/month: log search alert rules at 15-minute frequency are $0.50 a rule a
// month (Azure Retail Prices API, Central India, "Alerts System Log Monitored at
// 15 Minute Frequency", read 14 Oct 2026); emails are free for the first 1,000
// a month ("Emails"). Five-minute rules would be $1.50 each.
//
// WHERE TO LOOK (portal tour)
// Monitor > Alerts (fired alerts) · Monitor > Alerts > Alert rules (these six) ·