Eval Harness Tutorial · Architecture
When checks run: offline, inline, online
The same check can run at three different times. Which one you pick decides what the check can do about a problem.
- Inline runs during the turn, before the customer sees anything. It can block, redact or hand over. It must be fast, because the customer is waiting.
- Online runs after the turn, on real traffic, usually a sample. It can't undo a reply, but it can alert, open a review and spot trends. It can take seconds, so AI judges are fine here.
- Offline runs before release or in batch: on golden test sets in CI (a fail blocks the release), re-running new rules over old traces, and cost rules that need days of data.
| Offline | Inline | Online | |
|---|---|---|---|
| When | Before release, or in batch | During the turn | Seconds to minutes after the turn |
| Can it stop a bad reply? | No; it stops a bad release | Yes: block, redact, hand over | No; it alerts and opens a review |
| Time it may take | Minutes to hours | Milliseconds | Seconds |
| What it reads | Golden sets, months of traces, cost records | The request at that moment | Traces and whole sessions |
| Good for | Regression, cost rules, backfill | PII, injection, argument limits, spend caps | The five patterns, groundedness, judged checks |
| Built here | Not built yet release CLI: Tier 7; cost rules: Tier 5 | Not built yet any check at any moment: Tier 3 (the reference agent's harness already does it its own way) | Built from a trace file and App Insights |
Each check declares its speed
Every check states how fast it is, and the framework refuses to put a slow check in a fast place. Measured: DeepEval's “deterministic” check takes about 500 ms, so it's online, not inline; Presidio takes about 2 s on its first call, so an inline host must warm it up.
| Check | Provider | Speed class | Expected time |
|---|---|---|---|
argument_limit | ours | online | 5 ms |
azure_groundedness | azure | online | 3000 ms |
bedrock_guardrails | aws | inline | 400 ms |
claim_needs_action | ours | online | 5 ms |
consistency | ours | online | 5 ms |
decision_vs_should | ours | online | 5 ms |
deepeval_tool_correctness | deepeval | online | 500 ms |
presidio_pii | presidio | inline | 50 ms |
ragas_faithfulness | ragas | online | 6000 ms |
requires_before | ours | online | 5 ms |
Decided · Offline, inline, online
- Is the three-timings explanation clear enough for an ops lead deciding where a rule should run?
Decided: Yes. The comparison table is the one-slide version.