cleandataengineering.aiEval Harness, drawn simply
You are here: Architecture › Offline, inline, online

Eval Harness Tutorial · Architecture

When checks run: offline, inline, online

The same check can run at three different times. Which one you pick decides what the check can do about a problem.

  • Inline runs during the turn, before the customer sees anything. It can block, redact or hand over. It must be fast, because the customer is waiting.
  • Online runs after the turn, on real traffic, usually a sample. It can't undo a reply, but it can alert, open a review and spot trends. It can take seconds, so AI judges are fine here.
  • Offline runs before release or in batch: on golden test sets in CI (a fail blocks the release), re-running new rules over old traces, and cost rules that need days of data.
OFFLINE · before release, or in batchGolden test setknown questions + answersRelease gate in CIa fail blocks the mergeBackfill and scheduled re-runsnew rules on old tracesCost rules over daysmany calls, not oneINLINE · during the turn, before the customer sees anythingCustomersends a messageModel calldecides what to doTool runse.g. issue_refundModel callwrites the replyCustomerreads itor at the gateway (LiteLLM, APIM): it sees every model call1pre_modelwhat goes to the model2post_model3pre_toolbefore a tool acts4post_tool5replythe reply itselfONLINE · after the turn, on live trafficTrace writtenby the agent's own tracingWhere traces landPhoenix, Langfuse, App Insightspull or tapEval workera sample of traffic; session ruleswhen a conversation endsScores and alertsinto your tools
The five numbered points are the inline moments; each can also be reached at the gateway for model calls. Online and offline read what the agent already recorded.
OfflineInlineOnline
WhenBefore release, or in batchDuring the turnSeconds to minutes after the turn
Can it stop a bad reply?No; it stops a bad releaseYes: block, redact, hand overNo; it alerts and opens a review
Time it may takeMinutes to hoursMillisecondsSeconds
What it readsGolden sets, months of traces, cost recordsThe request at that momentTraces and whole sessions
Good forRegression, cost rules, backfillPII, injection, argument limits, spend capsThe five patterns, groundedness, judged checks
Built hereNot built yet release CLI: Tier 7; cost rules: Tier 5Not built yet any check at any moment: Tier 3 (the reference agent's harness already does it its own way)Built from a trace file and App Insights

Each check declares its speed

Every check states how fast it is, and the framework refuses to put a slow check in a fast place. Measured: DeepEval's “deterministic” check takes about 500 ms, so it's online, not inline; Presidio takes about 2 s on its first call, so an inline host must warm it up.

CheckProviderSpeed classExpected time
argument_limitoursonline5 ms
azure_groundednessazureonline3000 ms
bedrock_guardrailsawsinline400 ms
claim_needs_actionoursonline5 ms
consistencyoursonline5 ms
decision_vs_shouldoursonline5 ms
deepeval_tool_correctnessdeepevalonline500 ms
presidio_piipresidioinline50 ms
ragas_faithfulnessragasonline6000 ms
requires_beforeoursonline5 ms
Decided · Offline, inline, online
  1. Is the three-timings explanation clear enough for an ops lead deciding where a rule should run?
    Decided: Yes. The comparison table is the one-slide version.