Agent Harness Tutorial · Tech stack
L11 · Observability
Traces, metrics and alerts for every step. Family: See · around every turn.
Why our choiceAn Azure managed stack fed by OTel, so swapping App Insights for Langfuse touches one adapter. Sampling stays at 100%: a sampled-out span is one the evals never see.
| Option | Where from | Verdict | Why |
|---|---|---|---|
| OpenTelemetry | ours | ✓ chosen | behind one helper |
| azure-monitor-opentelemetry | ours | ✓ chosen | into App Insights, 100% sampling |
| alerts.bicep | ours | ✓ chosen | log alerts |
| Application Insights | Azure | ✓ used | |
| Managed Prometheus + Grafana | Azure | ◐ planned | tier 12 |
| Foundry tracing | Azure | · considered | |
| OpenTelemetry | open source | ✓ used | the standard |
| Langfuse · Prometheus · Grafana | open source | · considered | reference agent, local |
| Jaeger | open source | · considered |
How it works inside the harness: The trace.
Compared: Where do traces, metrics and alerts go?
| Option | From | Verdict | Holds | Where used | Customer's words | Alerts | Status |
|---|---|---|---|---|---|---|---|
| Application Insights (OTel distro) | Azure | chosen | spans: one story per conversation | Azure | not captured (privacy) | log alerts, budget alert | on Azure today |
| Managed Prometheus + Grafana | Azure | planned | counters, alert rules, dashboards | Azure | never | same rules as reference | planned, Tier 12 |
| Langfuse | open source | reference agent | traces and scores | reference agent, locally | on sampled turns, redacted | via the watch job | second exporter option |
| Prometheus + Grafana (self-run) | open source | reference agent | counters, 21 alerts, dashboard | reference agent, locally | never | Alertmanager | Docker compose |
| Foundry tracing view | Azure | with Foundry | agent traces in Foundry | Azure | not decided | no | comes with a Foundry model |
| OTel Collector + Jaeger | open source | alternative | spans, routed anywhere | anywhere | as configured | no | listed open source option |
| LangSmith | vendor | not used | traces, datasets (paid) | with LangGraph | as configured | yes | not used |
DecisionSpans go to Application Insights and counters to Managed Prometheus, both fed by one OpenTelemetry setup. Langfuse plays the trace-store role in the reference agent and could be added as a second exporter; swapping is one adapter and one YAML line.
How it is done: the real code
Cut from the live repositories each time the site is built, so it always matches what runs.
telemetry:
adapter: azure-monitor-otel
connection_string: {key_vault: appinsights-connection-string}// WHAT IT IS
// Email alerts on the agent's own telemetry: one action group (email to the
// owner) and six log alert rules, each a KQL query over the Log Analytics
// workspace App Insights writes to, run every 15 minutes. A rule fires when its
// query returns any row in the last 15 minutes:
// turn-failed a turn ended Failed ("I am having trouble answering")
// agent-not-starting a container exited with code 1, or printed a traceback
// (F-118: the new revision failed and nobody knew)
// model-calls-failing APIM answered a model call with 4xx/5xx, or the agent's
// model call failed
// content-safety Content Safety blocked a prompt or completion, or was
// unavailable (the F0 quota spent: calls pass unscreened)
// person-needed a hand-off was raised, or a payout is waiting for a
// claims handler — someone has to act
// check-blocked an inline check blocked (injection, PII, a promise of
// cover, money in £/$/€…): an attack or a rule misfiring
//
// WHICH CONCERN IT SERVES
// Observability (deck: rule → finding → alert; PLAN Tier 4b B3 "log alerts").
// Until this, App Insights only collected: the $10 budget was the only alert.
// The online watch rules (findings table, Workbook, canary) are still Tier 4b.
//
// EXPECTED DEV COST
// ~$3/month: log search alert rules at 15-minute frequency are $0.50 a rule a
// month (Azure Retail Prices API, Central India, "Alerts System Log Monitored at
// 15 Minute Frequency", read 14 Oct 2026); emails are free for the first 1,000
// a month ("Emails"). Five-minute rules would be $1.50 each.
//
// WHERE TO LOOK (portal tour)
// Monitor > Alerts (fired alerts) · Monitor > Alerts > Alert rules (these six) ·