Agent Harness Tutorial · End to end · Zoom level 2
How a check is written
A check is two things: a small function, and one line in a file that says where it runs and what a failure does. To move a check, or to change what happens when it fails, you only edit the file.
The rule returns a verdict: Verdict(allowed, rule, reason). The placement file looks like this (shortened):
pre_model: [{use: no_known_injection, on_fail: safe_reply}]
post_tool: [{use: no_instructions_in_result, on_fail: hold_writes}]
reply: 7 rules, each on_fail: safe_reply
online: sample: 1.0, 5 checks, on_fail: alert # re-run on traces
release: tool_selection, must_include, on_fail: block, min: 1.0
The same file covers all three times a check can run: inside the turn, online on traces, and at release. That is the link between checks and eval: one rule library, three speeds.
Every check writes an agent.policy span, and a counter of refusals per rule, so we can see how often each one fires.
How it is done: the real code
Cut from the live repositories each time the site is built, so it always matches what runs.
claims_fnol/policy/evaluators.yaml:11The whole placement file: every check, where it runs, what a fail doesevaluators:
no_unclaimed_effect: {kind: rule}
no_promise_of_cover: {kind: rule} # P-NO-PROMISE
no_ungrounded_entity: {kind: rule}
no_superseded_state: {kind: rule} # the harness's, AHC-0117
no_superseded_claim_state: {kind: rule} # the same, clause by clause (F-4)
no_pii_echo: {kind: rule}
money_in_rupees: {kind: rule} # F-25: never £, $ or €
tool_selection: {kind: rule} # the library's: a golden case's must-call
must_include: {kind: rule} # the library's: a golden case's must-include
# A9: APIM's Content Safety verdict on each model call (the x-content-safety
# header), recorded as a check result. It judges nothing again and never
# blocks: APIM already blocked. Skips locally, where there is no APIM.
content_safety: {kind: guardrail_log, source: x-content-safety}
# A11, the library's (agent_harness.evals.injection): known injection phrasing
# in the policyholder's words, and instructions inside a tool's result.
no_known_injection: {kind: rule}
no_instructions_in_result: {kind: rule}
positions:
# Inline, before the model is paid for: a fail ends the turn with the safe
# reply, the refusal the loop already gives at PRE_MODEL (A11).
pre_model:
- {use: no_known_injection, on_fail: safe_reply}
# Inline, on each tool result before it enters context: the result still goes
# in, fenced as data like every result; any write later in the same run is
# refused until the policyholder confirms it in a message of their own (A11).
post_tool:
- {use: no_instructions_in_result, on_fail: hold_writes}
# Inline, before the policyholder reads it: a fail is the safe reply. Each runs
# on the model's reply; those that need nothing but the reply also run on every
# route's (a template's), where there are no tool results to ground against.