Eval Harness Tutorial · From traces
A real agent's traces: FNOL in App Insights
Until now every check ran on recorded test conversations. This page is the
first run on a deployed agent: FNOL's traces, read from its App Insights in
Azure, with read-only access. The numbers below come from that run (counts only,
saved in docs/runs/).
What was there
| Found | |
|---|---|
| Spans read | 688 |
| Turns | 39 (13 Oct 2026 to 14 Oct 2026, one test user) |
| Turns with the conversation's words | 0: content capture is off |
| Turns with a conversation id | 0: the deployed build predates the fix that adds it |
| How turns ended | completed: 28 refused/R-COVERAGE: 1 refused/no_known_injection: 2 refused/agent.paused: 1 needs_approval: 2 escalated/injury-reported: 2 escalated/turns-exceeded: 2 escalated/asked-for-human: 1 |
| Tool calls | get_claim ok: 16 get_claim refused: 1 get_policy ok: 6 issue_payout ok: 2 list_claims ok: 2 list_policies ok: 2 register_claim ok: 2 request_payout unrecorded: 3 submit_document ok: 2 withdraw_claim ok: 1 |
The lesson: missing evidence must never become a verdict
The first run reported two over-escalations: payouts sent for approval “with no rule calling for it”. They were false alarms. Those payouts were sent for approval because the claim was over ₹25,000, but the claim's amount wasn't in the traces, because content capture is off. The rule found no amount and read that as “not over ₹25,000”. The hand-over rule had the same flaw: it passed turns whose customer words it never saw.
The fix, now in the code:
- A session says what its source didn't record:
unrecorded= text, tool arguments, tool results. - Each condition in a rule has three answers: yes, no, or unknown, when it needs something unrecorded that could have changed the answer.
- A check that can't judge says skip: could not judge, with the reason. A partly judged one says how much it couldn't judge.
Structure is still judged: “two tool calls failed in a row” or “a payout with no claim read at all” need no words, so those checks still pass or fail.
The rules on FNOL's real traces, after the fix
| Rule | Pattern | Results |
|---|---|---|
payout_needs_claim_lookup | requires_before | nothing to check: 37, skip: 2 |
payout_approval | decision_vs_should | pass: 36, skip: 3 |
payout_promise_kept | claim_needs_action | skip: 39 |
handed_to_person | decision_vs_should | skip: 34, pass: 5 |
facts_from_systems | consistency | skip: 39 |
| Saved question | Rows |
|---|---|
under_escalations | 0 |
over_escalations | 0 |
promises_not_kept | 0 |
actions_without_prerequisite | 0 |
limits_exceeded | 0 |
unsupported_facts | 0 |
A production agent with privacy on records what happened (tools, outcomes, endings) but not what was said. Reading its traces afterwards can check structure, not content. Content checks (promises, facts, the customer's words) have to run where the content is: inside the agent's own environment, inline or just after the turn, with only the verdicts sent to App Insights. That's the next step.
- For content checks on a production agent, where should they run?
Decided: Inside the agent's environment, where the words already are; only verdicts go to App Insights, so no customer words leave. Content capture stays off in production (T30, 19 Oct). Built: see the next page.
Code: evalharness/sources/app_insights.py, scripts/pull_app_insights.py,
scripts/checks_on_traces.py. Where checks can run: offline, inline, online. Built App Insights source