cleandataengineering.aiAgent Harness, drawn simply
You are here: End to end › Checks › How a check is written

Agent Harness Tutorial · End to end · Zoom level 2

How a check is written

A check is two things: a small function, and one line in a file that says where it runs and what a failure does. To move a check, or to change what happens when it fails, you only edit the file.

A rulecontext in,verdict outevaluators.yamlnames each check,places it at a point,says what a fail doesChecked at start-upunknown name? slowerthan 50 ms? → refuse
The file is checked when the app starts. A check placed where its inputs never exist, or one that is too slow for its point, stops the app from starting.

The rule returns a verdict: Verdict(allowed, rule, reason). The placement file looks like this (shortened):

pre_model: [{use: no_known_injection,        on_fail: safe_reply}]
post_tool: [{use: no_instructions_in_result, on_fail: hold_writes}]
reply:     7 rules, each on_fail: safe_reply
online:    sample: 1.0, 5 checks, on_fail: alert     # re-run on traces
release:   tool_selection, must_include, on_fail: block, min: 1.0

The same file covers all three times a check can run: inside the turn, online on traces, and at release. That is the link between checks and eval: one rule library, three speeds.

Every check writes an agent.policy span, and a counter of refusals per rule, so we can see how often each one fires.

How it is done: the real code

Cut from the live repositories each time the site is built, so it always matches what runs.

claims_fnol/policy/evaluators.yaml:11The whole placement file: every check, where it runs, what a fail does
evaluators:
  no_unclaimed_effect: {kind: rule}
  no_promise_of_cover: {kind: rule}        # P-NO-PROMISE
  no_ungrounded_entity: {kind: rule}
  no_superseded_state: {kind: rule}        # the harness's, AHC-0117
  no_superseded_claim_state: {kind: rule}  # the same, clause by clause (F-4)
  no_pii_echo: {kind: rule}
  money_in_rupees: {kind: rule}            # F-25: never £, $ or €
  tool_selection: {kind: rule}             # the library's: a golden case's must-call
  must_include: {kind: rule}               # the library's: a golden case's must-include
  # A9: APIM's Content Safety verdict on each model call (the x-content-safety
  # header), recorded as a check result. It judges nothing again and never
  # blocks: APIM already blocked. Skips locally, where there is no APIM.
  content_safety: {kind: guardrail_log, source: x-content-safety}
  # A11, the library's (agent_harness.evals.injection): known injection phrasing
  # in the policyholder's words, and instructions inside a tool's result.
  no_known_injection: {kind: rule}
  no_instructions_in_result: {kind: rule}

positions:
  # Inline, before the model is paid for: a fail ends the turn with the safe
  # reply, the refusal the loop already gives at PRE_MODEL (A11).
  pre_model:
    - {use: no_known_injection, on_fail: safe_reply}

  # Inline, on each tool result before it enters context: the result still goes
  # in, fenced as data like every result; any write later in the same run is
  # refused until the policyholder confirms it in a message of their own (A11).
  post_tool:
    - {use: no_instructions_in_result, on_fail: hold_writes}

  # Inline, before the policyholder reads it: a fail is the safe reply. Each runs
  # on the model's reply; those that need nothing but the reply also run on every
  # route's (a template's), where there are no tool results to ground against.