cleandataengineering.aiAgent Harness, drawn simply
You are here: End to end › Checks

Agent Harness Tutorial · End to end · Zoom level 1

Checks at every point

A check is a small piece of code that looks at one thing (a message, an answer, a tool request, a result) and says pass or block. The harness has checks at eight points along a turn, because each kind of mistake can only be seen at one place.

Goes inWhatever is passing that point
What happensRun the checks placed there; the first block wins; a check that crashes also blocks
Comes outPass, or a named block with a safe next step
Open: 11before loopopen ›Open: 22before modelopen ›Open: 33after modelopen ›Open: 44before toolopen ›Open: 55the toolopen ›Open: 66after toolopen ›Open: 77after loopopen ›Open: 88before replyopen ›ONE TURN, LEFT TO RIGHTinside the loop: 2 to 6 repeat on every tripRules~1 ms, exact, freeClassic ML / small modelsms to 100s of msLLM judgesseconds and money; offline only
Click a number to open that point. Points 2 to 6 sit inside the loop, so they run again on every trip.

Almost every check is a plain rule. Rules are instant, free and give the same answer every time, so they can block safely in the middle of a turn. Azure adds three: a model allow-list, a rate limit, and Content Safety. No LLM judge runs inside a turn; judges are slow and costly, so they belong in offline eval.

What a block does depends on where it happens. Before or after the model, or before the reply, the turn ends with a safe reply: "I am not able to confirm that. Let me pass you to a colleague who can help." Before a tool, only that one call is refused, and the model is told why so it can choose again.

Fail closed. If a check itself crashes, that counts as a block. A broken guard must never mean an open door.

Who provides the checksOurs (harness and agent): about 30, all built, including every check that protects money. Azure APIM: 3. Pydantic AI: none (it only turns errors into types). Open source to add: Prompt Guard 2, and maybe Presidio for personal data.

Want every detail? The long version is in the Series: policy at every position, with the live file.

Code view: input, process, output, the call stack, the classes

Goes inWhat is about to happen: the customer's words, a planned call, or a result
What happensThe position's rules run in order inside an agent.policy span; the first block wins and a rule that raises also blocks
Comes outA Verdict: allowed, or blocked with the rule and reason
CALL STACK · START TO FINISHGOES IN → COMES OUT1Open the source: claims_fnol/policy/__init__.py:205PLAN = evaluators.load(EVALUATORS, rules=RULES)claims_fnol/policy/__init__.py:205in: evaluators.yaml, at importout: the Plan, or PlanRefused stops startup2Open the source: agent_harness/evals/plan.py:232load(source, *, rules, kinds=KINDS)agent_harness/evals/plan.py:232in: YAML path and the agent's rule catalogueout: a checked Plan3Open the source: agent_harness/loop/__init__.py:244_Run._ask(step_span)agent_harness/loop/__init__.py:244in: the transcript before the model callout: a ModelResponse, or a refused Ended4Open the source: agent_harness/loop/screen.py:40Screen.at(Position.PRE_MODEL, results, text=)agent_harness/loop/screen.py:40in: the customer's latest wordsout: a Verdict; blocked_by set on run span5Open the source: agent_harness/policy/__init__.py:63enforce(ctx, rules)agent_harness/policy/__init__.py:63in: a policy Context and the position's rulesout: first blocking Verdict, or ALLOW6Open the source: agent_harness/evals/screens.py:99_judging(...).check(ctx)agent_harness/evals/screens.py:99in: the Context (rule no_known_injection)out: Verdict from the evaluator's result7Open the source: agent_harness/evals/__init__.py:201judge(evaluator, request)agent_harness/evals/__init__.py:201in: an EvalRequest built by asked(ctx)out: EvalResult, or skipped if needs unmet8Open the source: agent_harness/evals/rule.py:90RuleEvaluator.evaluate(request)agent_harness/evals/rule.py:90in: the requestout: pass / fail / error EvalResult9Open the source: agent_harness/evals/injection.py:110no_known_injection(request)agent_harness/evals/injection.py:110in: the query textout: Verdict on known injection phrases10Open the source: agent_harness/loop/__init__.py:354_Run._act(planned)agent_harness/loop/__init__.py:354in: the planned tool callsout: results fed back, or an Ended11Open the source: agent_harness/loop/screen.py:52Screen.permitted(planned, results)agent_harness/loop/screen.py:52in: each call, at Position.PRE_TOOLout: refused calls as error results12Open the source: claims_fnol/policy/__init__.py:228policyholder_asked(ctx)claims_fnol/policy/__init__.py:228in: tool name, record id, identity.consentedout: ALLOW, or block if not asked for13Open the source: agent_harness/policy/verdicts.py:80block(rule, reason)agent_harness/policy/verdicts.py:80in: the rule name and reasonout: Verdict(allowed=False, ...)14Open the source: agent_harness/loop/screen.py:79Screen.admitted(call, result, results)agent_harness/loop/screen.py:79in: one result, at Position.POST_TOOLout: the result, or 'withheld by <rule>'
Read top to bottom: each box is one function call, indented under the function that called it; the dark box is where it starts. Click a box to open its source file.
Dashed boxes are protocols (the ports) or enums; grey boxes are data models; white boxes are classes with behaviour. A dashed arrow means "implements". Click a class to open its source.

How it is done: the real code

Cut from the live repositories each time the site is built, so it always matches what runs.

agent_harness/policy/__init__.py:63enforce(): run the rules for one point; the first block wins
def enforce(ctx: Context, rules: Sequence[Rule] | None = None) -> Verdict:
    """Run the rules for this position. First block wins.

    A rule that raises **blocks**. That is the whole of AAC-0091: a guardrail
    that errors open is believed and absent at once, and nobody goes looking for
    a control they think they have.
    """
    applicable = default_rules().get(ctx.position, ()) if rules is None else rules
    with tel.span("agent.policy", **{"agent.policy.position": ctx.position.value}) as span:
        for rule in applicable:
            name = getattr(rule, "__name__", repr(rule))
            try:
                verdict = rule(ctx)
            except Exception as exc:  # noqa: BLE001 — deliberate: fail closed
                span.set_attribute("agent.policy.blocked_by", name)
                span.set_attribute("agent.policy.errored", True)
                return block(name, f"rule failed and traffic was blocked: {exc}")
            if verdict.blocked:
                span.set_attribute("agent.policy.blocked_by", verdict.rule or name)
                return verdict
        return ALLOW
agent_harness/policy/__init__.py:94What Rohan gets when a check blocks
SAFE_REPLY = "I am not able to confirm that. Let me pass you to a colleague who can help."
claims_fnol/policy/__init__.py:246Which rules run at which point, for this agent
DEFAULT_RULES: dict[Position, tuple[Rule, ...]] = {
    # A11: known injection phrasing in the policyholder's words, before the model.
    Position.PRE_MODEL: PLAN.at(Position.PRE_MODEL),
    Position.POST_MODEL: OUTPUT_RULES,
    # P-OWN-WORDS first; then A11's hold: no write after a result carrying instructions.
    Position.PRE_TOOL: (policyholder_asked, *PLAN.at(Position.PRE_TOOL)),
    Position.POST_TOOL: PLAN.at(Position.POST_TOOL),
    Position.REPLY: REPLY_RULES,
}

Tech choices (what we chose, Azure and open-source options): L7 · Policy enforcement

Look inside