Agent Harness Tutorial · End to end · Zoom level 2
Inline eval
Inline eval is the checks inside the turn. They have their own section, Checks at every point, so this page only says how they fit into eval.
Inline checks must be fast (each under 50 ms, or the app refuses to start), exact and free, because they sit on the path of every reply. That is why they are all rules today, and why LLM judges are kept offline.
Every block is written to the trace as an agent.policy span, so online eval can count blocks per rule and notice if one rule suddenly refuses far more than usual.
How it is done: the real code
Cut from the live repositories each time the site is built, so it always matches what runs.
claims_fnol/policy/evaluators.yaml:30evaluators.yaml: the inline positionspositions: # Inline, before the model is paid for: a fail ends the turn with the safe # reply, the refusal the loop already gives at PRE_MODEL (A11). pre_model: - {use: no_known_injection, on_fail: safe_reply} # Inline, on each tool result before it enters context: the result still goes # in, fenced as data like every result; any write later in the same run is # refused until the policyholder confirms it in a message of their own (A11). post_tool: - {use: no_instructions_in_result, on_fail: hold_writes} # Inline, before the policyholder reads it: a fail is the safe reply. Each runs # on the model's reply; those that need nothing but the reply also run on every # route's (a template's), where there are no tool results to ground against. reply: - {use: no_unclaimed_effect, on_fail: safe_reply} - {use: no_promise_of_cover, on_fail: safe_reply} - {use: no_ungrounded_entity, on_fail: safe_reply} - {use: no_superseded_state, on_fail: safe_reply} - {use: no_superseded_claim_state, on_fail: safe_reply} - {use: no_pii_echo, on_fail: safe_reply} - {use: money_in_rupees, on_fail: safe_reply} # After the turn, on what the policyholder was actually sent: a fail is a finding. online: