cleandataengineering.aiEval Harness, drawn simply
You are here: From traces › Real traces: FNOL

Eval Harness Tutorial · From traces

A real agent's traces: FNOL in App Insights

Until now every check ran on recorded test conversations. This page is the first run on a deployed agent: FNOL's traces, read from its App Insights in Azure, with read-only access. The numbers below come from that run (counts only, saved in docs/runs/).

App InsightsFNOL's deployed traces(Azure)KQLAppInsights sourceevalharness: rowsbecome spansOTLP file copyread anywhere, evenwithout the Azure SDKFNOL's own rebuilderturns and conversations,then the rules
App Insights to spans to a file copy to FNOL's own rebuilder. Only spans named agent.* and gen_ai.* are read, so HTTP calls (and anything in their URLs) never leave Azure.

What was there

Found
Spans read688
Turns39 (13 Oct 2026 to 14 Oct 2026, one test user)
Turns with the conversation's words0: content capture is off
Turns with a conversation id0: the deployed build predates the fix that adds it
How turns endedcompleted: 28
refused/R-COVERAGE: 1
refused/no_known_injection: 2
refused/agent.paused: 1
needs_approval: 2
escalated/injury-reported: 2
escalated/turns-exceeded: 2
escalated/asked-for-human: 1
Tool callsget_claim ok: 16
get_claim refused: 1
get_policy ok: 6
issue_payout ok: 2
list_claims ok: 2
list_policies ok: 2
register_claim ok: 2
request_payout unrecorded: 3
submit_document ok: 2
withdraw_claim ok: 1

The lesson: missing evidence must never become a verdict

The first run reported two over-escalations: payouts sent for approval “with no rule calling for it”. They were false alarms. Those payouts were sent for approval because the claim was over ₹25,000, but the claim's amount wasn't in the traces, because content capture is off. The rule found no amount and read that as “not over ₹25,000”. The hand-over rule had the same flaw: it passed turns whose customer words it never saw.

The fix, now in the code:

  1. A session says what its source didn't record: unrecorded = text, tool arguments, tool results.
  2. Each condition in a rule has three answers: yes, no, or unknown, when it needs something unrecorded that could have changed the answer.
  3. A check that can't judge says skip: could not judge, with the reason. A partly judged one says how much it couldn't judge.

Structure is still judged: “two tool calls failed in a row” or “a payout with no claim read at all” need no words, so those checks still pass or fail.

The rules on FNOL's real traces, after the fix

RulePatternResults
payout_needs_claim_lookuprequires_beforenothing to check: 37, skip: 2
payout_approvaldecision_vs_shouldpass: 36, skip: 3
payout_promise_keptclaim_needs_actionskip: 39
handed_to_persondecision_vs_shouldskip: 34, pass: 5
facts_from_systemsconsistencyskip: 39
Saved questionRows
under_escalations0
over_escalations0
promises_not_kept0
actions_without_prerequisite0
limits_exceeded0
unsupported_facts0
What this means for the product

A production agent with privacy on records what happened (tools, outcomes, endings) but not what was said. Reading its traces afterwards can check structure, not content. Content checks (promises, facts, the customer's words) have to run where the content is: inside the agent's own environment, inline or just after the turn, with only the verdicts sent to App Insights. That's the next step.

Decided · Real traces
  1. For content checks on a production agent, where should they run?
    Decided: Inside the agent's environment, where the words already are; only verdicts go to App Insights, so no customer words leave. Content capture stays off in production (T30, 19 Oct). Built: see the next page.

Code: evalharness/sources/app_insights.py, scripts/pull_app_insights.py, scripts/checks_on_traces.py. Where checks can run: offline, inline, online. Built App Insights source