cleandataengineering.aiAgent Harness, drawn simply
You are here: End to end › Eval

Agent Harness Tutorial · End to end · Zoom level 1

Eval: proving it works

An agent can be wrong without any error at all. "Yes, your claim is being processed," said confidently, when no claim was ever created: unit tests pass, latency is fine, nothing turns red. Eval is how we catch that. Every turn leaves a record, and we read that record at three speeds.

Goes inThe trace of every turn, plus test scenarios
What happensCheck inside the turn, re-check on traces minutes later, and run whole scenarios before each release
Comes outBlocked bad replies, alerts on what slipped through, and releases that can't get worse
Same record, three readers. Inline prevents, online detects, offline stops a bad release.
SpeedWhenCan it change the reply?Built on Azure?
Inlineinside the turnYes: it blocksYes, these are the 8 check points
Onlineminutes after, on stored tracesNo, the reply has gone; it alertsPartly: alerts yes, the rule job not yet
Offlinein CI, before every releaseNo: it blocks the releaseYes: 29 scenarios, 750 tests

Want every detail? The long version is in the Series: observability on Azure, what runs today.

How it is done: the real code

Cut from the live repositories each time the site is built, so it always matches what runs.

claims_fnol/policy/evaluators.yaml:55evaluators.yaml: the same checks, online and at release
  online:
    sample: 1.0
    run:
      - {use: no_ungrounded_entity, on_fail: alert}
      - {use: money_in_rupees, on_fail: alert}
      - {use: content_safety, on_fail: alert}   # A9: online only, never at reply (F-79)
      - {use: no_known_injection, on_fail: alert}
      - {use: no_instructions_in_result, on_fail: alert}

  # Over golden cases and scenarios: a pass rate under `min` blocks the release.
  release:
    run:
      - {use: tool_selection, on_fail: block, min: 1.0}
      - {use: must_include, on_fail: block, min: 1.0}

Tech choices (what we chose, Azure and open-source options): L12 · Eval harness

Look inside