Agent Harness Tutorial · End to end · Zoom level 1
Eval: proving it works
An agent can be wrong without any error at all. "Yes, your claim is being processed," said confidently, when no claim was ever created: unit tests pass, latency is fine, nothing turns red. Eval is how we catch that. Every turn leaves a record, and we read that record at three speeds.
Goes inThe trace of every turn, plus test scenarios
What happensCheck inside the turn, re-check on traces minutes later, and run whole scenarios before each release
Comes outBlocked bad replies, alerts on what slipped through, and releases that can't get worse
| Speed | When | Can it change the reply? | Built on Azure? |
|---|---|---|---|
| Inline | inside the turn | Yes: it blocks | Yes, these are the 8 check points |
| Online | minutes after, on stored traces | No, the reply has gone; it alerts | Partly: alerts yes, the rule job not yet |
| Offline | in CI, before every release | No: it blocks the release | Yes: 29 scenarios, 750 tests |
Want every detail? The long version is in the Series: observability on Azure, what runs today.
How it is done: the real code
Cut from the live repositories each time the site is built, so it always matches what runs.
claims_fnol/policy/evaluators.yaml:55evaluators.yaml: the same checks, online and at release online:
sample: 1.0
run:
- {use: no_ungrounded_entity, on_fail: alert}
- {use: money_in_rupees, on_fail: alert}
- {use: content_safety, on_fail: alert} # A9: online only, never at reply (F-79)
- {use: no_known_injection, on_fail: alert}
- {use: no_instructions_in_result, on_fail: alert}
# Over golden cases and scenarios: a pass rate under `min` blocks the release.
release:
run:
- {use: tool_selection, on_fail: block, min: 1.0}
- {use: must_include, on_fail: block, min: 1.0}Tech choices (what we chose, Azure and open-source options): L12 · Eval harness
Look inside
The traceOne trace per turn: a tree of spans, with no customer words in it.Inline evalThe 8 check points: fast, exact, able to block.Online evalRe-read finished turns, find what slipped through, raise an alert.Offline evalBefore every release: scenarios, replayed model calls, a gate.The rule libraryFour kinds of check, one plug-in shape.Improve: the miss becomes a testDetect, troubleshoot, recover, remember.