cleandataengineering.aiAgent Harness, drawn simply
You are here: Tech stack › L11 · Observability

Agent Harness Tutorial · Tech stack

L11 · Observability

Traces, metrics and alerts for every step. Family: See · around every turn.

AZURE OPTIONSApplication Insights✓ usedManaged Prometheus + Grafana◐ planned: tier 12Foundry tracing· consideredOUR CHOICEOpenTelemetrybehind one helperazure-monitor-opentelemetryinto App Insights, 100% samplingalerts.biceplog alertsOPEN-SOURCE OPTIONSOpenTelemetry✓ used: the standardLangfuse · Prometheus · Grafana· considered: reference agent, localJaeger· consideredWHO DOES THE WORKAzure gives 55% · we code once 35% · each agent team still does 10%
Dark boxes are what we run. Solid boxes are used or planned; dashed boxes were considered or dropped (crossed). The bar: grey is what Azure gives, dark is what we code once, hatched is what each agent team writes.
Why our choiceAn Azure managed stack fed by OTel, so swapping App Insights for Langfuse touches one adapter. Sampling stays at 100%: a sampled-out span is one the evals never see.
OptionWhere fromVerdictWhy
OpenTelemetryours✓ chosenbehind one helper
azure-monitor-opentelemetryours✓ choseninto App Insights, 100% sampling
alerts.bicepours✓ chosenlog alerts
Application InsightsAzure✓ used
Managed Prometheus + GrafanaAzure◐ plannedtier 12
Foundry tracingAzure· considered
OpenTelemetryopen source✓ usedthe standard
Langfuse · Prometheus · Grafanaopen source· consideredreference agent, local
Jaegeropen source· considered

How it works inside the harness: The trace.

Compared: Where do traces, metrics and alerts go?

Application In…AzurechosenManaged Promet…AzureplannedLangfuseopen sourcereference agentPrometheus + G…open sourcereference agentFoundry tracin…Azurewith FoundryOTel Collector…open sourcealternativeLangSmithvendornot used
Dark: what we chose. Crossed and dashed: rejected or not for us. From the Series.
OptionFromVerdictHoldsWhere usedCustomer's wordsAlertsStatus
Application Insights (OTel distro)Azurechosenspans: one story per conversationAzurenot captured (privacy)log alerts, budget alerton Azure today
Managed Prometheus + GrafanaAzureplannedcounters, alert rules, dashboardsAzureneversame rules as referenceplanned, Tier 12
Langfuseopen sourcereference agenttraces and scoresreference agent, locallyon sampled turns, redactedvia the watch jobsecond exporter option
Prometheus + Grafana (self-run)open sourcereference agentcounters, 21 alerts, dashboardreference agent, locallyneverAlertmanagerDocker compose
Foundry tracing viewAzurewith Foundryagent traces in FoundryAzurenot decidednocomes with a Foundry model
OTel Collector + Jaegeropen sourcealternativespans, routed anywhereanywhereas configurednolisted open source option
LangSmithvendornot usedtraces, datasets (paid)with LangGraphas configuredyesnot used
DecisionSpans go to Application Insights and counters to Managed Prometheus, both fed by one OpenTelemetry setup. Langfuse plays the trace-store role in the reference agent and could be added as a second exporter; swapping is one adapter and one YAML line.

How it is done: the real code

Cut from the live repositories each time the site is built, so it always matches what runs.

  telemetry:
    adapter: azure-monitor-otel
    connection_string: {key_vault: appinsights-connection-string}
// WHAT IT IS
//   Email alerts on the agent's own telemetry: one action group (email to the
//   owner) and six log alert rules, each a KQL query over the Log Analytics
//   workspace App Insights writes to, run every 15 minutes. A rule fires when its
//   query returns any row in the last 15 minutes:
//     turn-failed           a turn ended Failed ("I am having trouble answering")
//     agent-not-starting    a container exited with code 1, or printed a traceback
//                           (F-118: the new revision failed and nobody knew)
//     model-calls-failing   APIM answered a model call with 4xx/5xx, or the agent's
//                           model call failed
//     content-safety        Content Safety blocked a prompt or completion, or was
//                           unavailable (the F0 quota spent: calls pass unscreened)
//     person-needed         a hand-off was raised, or a payout is waiting for a
//                           claims handler — someone has to act
//     check-blocked         an inline check blocked (injection, PII, a promise of
//                           cover, money in £/$/€…): an attack or a rule misfiring
//
// WHICH CONCERN IT SERVES
//   Observability (deck: rule → finding → alert; PLAN Tier 4b B3 "log alerts").
//   Until this, App Insights only collected: the $10 budget was the only alert.
//   The online watch rules (findings table, Workbook, canary) are still Tier 4b.
//
// EXPECTED DEV COST
//   ~$3/month: log search alert rules at 15-minute frequency are $0.50 a rule a
//   month (Azure Retail Prices API, Central India, "Alerts System Log Monitored at
//   15 Minute Frequency", read 14 Oct 2026); emails are free for the first 1,000
//   a month ("Emails"). Five-minute rules would be $1.50 each.
//
// WHERE TO LOOK (portal tour)
//   Monitor > Alerts (fired alerts) · Monitor > Alerts > Alert rules (these six) ·