cleandataengineering.aiAgent Harness, drawn simply
You are here: Cost › Who checks it

Agent Harness Tutorial · Cost

Who checks cost, and when

Cost is checked at six speeds, just like quality in eval: inside the turn, minutes later, at every release, daily, and monthly.

Inline · every callthe loop's Meterbuiltturn spend, steps, time, tool calls→ the run stopsOnline · minuteswatch job · alert rulesreference agent; not on Azure yetcostly turn over $0.05 · p95 cost per run vs last week→ trend · ticketReleasethe release gatebaseline builtcost per task vs the last release→ release blockedDailyLLM Cost Evaluation Frameworkv0: 6 rules, synthetic datawhy spend changed: 36 rules, savings per fix→ finding to the agent's ownerGateway · continuousAPIM token limitsto configuretokens per agent and customer tier→ 429 · alertMonthlyCost Management · FinOps$10 budget builtteam budgets, idle capacity→ budget email · review
Dashed boxes are designed or partly built; solid ones run.

The daily engine explains a change the way finance does: a baseline (median of the last 4 same weekdays), then a split into five drivers that add up exactly (volume, model mix, size, cache and tier, price), then a drill-down by agent, model and key. Example finding: "Up 23%, because traffic moved to a pricier model on the claims key." Every number comes from the engine; a model only polishes the wording.

Alerts. In the reference agent: SpendPerRunDoubled (p95 cost per run is twice last week's, at least 20 runs in the hour) and RunsEndingOnLimits (over 5% of runs ending on a budget). On Azure today only the $10 budget emails are about money.

How it is done: the real code

Cut from the live repositories each time the site is built, so it always matches what runs.

infra/modules/alerts.bicep:1The log alerts on Azure
// WHAT IT IS
//   Email alerts on the agent's own telemetry: one action group (email to the
//   owner) and six log alert rules, each a KQL query over the Log Analytics
//   workspace App Insights writes to, run every 15 minutes. A rule fires when its
//   query returns any row in the last 15 minutes:
//     turn-failed           a turn ended Failed ("I am having trouble answering")
//     agent-not-starting    a container exited with code 1, or printed a traceback
//                           (F-118: the new revision failed and nobody knew)
//     model-calls-failing   APIM answered a model call with 4xx/5xx, or the agent's
//                           model call failed
//     content-safety        Content Safety blocked a prompt or completion, or was
//                           unavailable (the F0 quota spent: calls pass unscreened)
//     person-needed         a hand-off was raised, or a payout is waiting for a
//                           claims handler — someone has to act
//     check-blocked         an inline check blocked (injection, PII, a promise of
//                           cover, money in £/$/€…): an attack or a rule misfiring
//
// WHICH CONCERN IT SERVES
//   Observability (deck: rule → finding → alert; PLAN Tier 4b B3 "log alerts").
//   Until this, App Insights only collected: the $10 budget was the only alert.
//   The online watch rules (findings table, Workbook, canary) are still Tier 4b.
//
// EXPECTED DEV COST
//   ~$3/month: log search alert rules at 15-minute frequency are $0.50 a rule a
//   month (Azure Retail Prices API, Central India, "Alerts System Log Monitored at
//   15 Minute Frequency", read 14 Oct 2026); emails are free for the first 1,000
//   a month ("Emails"). Five-minute rules would be $1.50 each.
//
// WHERE TO LOOK (portal tour)
//   Monitor > Alerts (fired alerts) · Monitor > Alerts > Alert rules (these six) ·