cleandataengineering.aiAgent Harness, drawn simply
You are here: End to end › Checks › How policies are implemented

Agent Harness Tutorial · End to end · Zoom level 2

How policies are implemented

A policy starts as one sentence a business owner agrees to, such as "a reply never says or implies that a claim is covered". It ends as a check that blocks a reply in 0.04 ms, a line in the trace when it fires, and a test that fails the release if anyone breaks it. Every step carries the same id, P-NO-PROMISE, so you can follow it end to end.

Goes inA sentence in the spec, with an id
What happensCode it once as a rule (or as an authority), place it with one YAML line, enforce it at a fixed point
Comes outA block in the turn, a span in the trace, a test that names the policy
Click any box to open its page. Both columns end with something a release gate can check.

Two shapes of policy. Some policies are about what may be said: those become rules that read the reply (left). Others are about who may do what: those become authority, built into which tools exist, a limit in configuration, a workflow that waits for a person, and a check at the far end (right). Neither is left to the model's judgement; the system prompt says the same things, but only as a first line of defence.

Policy (spec id)In wordsHow it is implementedWhere
P-NO-PROMISEnever say a claim is covered or will be paidreply rules no_promise_of_cover, no_ungrounded_entity; router refusal R-COVERAGEcheck point 3, 8, router
P-PAYOUTover ₹25,000 needs a claims handlerrequest_payout + DBOS approval + covers() at the claims systempayout flow
P-APPROVERnever approved by the policyholder whose claim it iscovers(): decided_by must differ from the holderapproval covers it
P-OWN-WORDSact only on what the policyholder asked forconsent from his own words + policyholder_asked before every writepolicyholder_asked
P-OWNERSHIPonly your own policies and claimsholder from the token; every query filtered by itonly his rows
P-WITHDRAWwithdraw until assessment startsthe claims system's precondition on withdraw_claimtool call

To add a policy: write the sentence in the spec with an id; write the rule once in RULES (or the authority in the spec and config); add one line to evaluators.yaml; write a test marked with the policy id. The app refuses to start if the YAML places a rule where its inputs never exist, or one slower than 50 ms at an inline point.

To change a policy: a limit lives in App Configuration (no deploy); a placement or an on-fail action lives in the YAML; only a new kind of rule needs code.

Code view: input, process, output, the call stack, the classes

Goes inevaluators.yaml and the insurer's rule catalogue RULES
What happensAt import the YAML is read and checked into a Plan, the Plan is turned into rules per position, and that table is registered as the default every check reads
Comes outDEFAULT_RULES, read by enforce() whenever no rules are passed
CALL STACK · START TO FINISHGOES IN → COMES OUT1Open the source: claims_fnol/policy/__init__.py:205PLAN = evaluators.load(EVALUATORS, rules=RULES)claims_fnol/policy/__init__.py:205in: evaluators.yaml beside the moduleout: the checked Plan2Open the source: agent_harness/evals/plan.py:232load(source, *, rules, kinds=KINDS)agent_harness/evals/plan.py:232in: the YAML pathout: Plan via parse()3Open the source: agent_harness/evals/plan.py:245parse(document, *, rules, kinds)agent_harness/evals/plan.py:245in: parsed YAML; RULES plus LIBRARY_RULESout: a Plan, or PlanRefused4Open the source: agent_harness/evals/plan.py:276_evaluator(name, spec, catalogue, kinds)agent_harness/evals/plan.py:276in: one entry under evaluators:out: an Evaluator built by its kind5Open the source: agent_harness/evals/rule.py:127RuleKind.build(name, spec, catalogue)agent_harness/evals/rule.py:127in: kind: rule and the catalogueout: a RuleEvaluator for that rule6Open the source: agent_harness/evals/plan.py:301_placed(position, entries, sample, built)agent_harness/evals/plan.py:301in: entries under positions.<name>out: a tuple of Placed7Open the source: agent_harness/evals/plan.py:316_checked(position, entry, evaluator, sample)agent_harness/evals/plan.py:316in: one use: entryout: Placed, or PlanRefused if it cannot run8Open the source: agent_harness/evals/plan.py:139Plan.inline(point, alert=None)agent_harness/evals/plan.py:139in: 'model' (line 209), 'every_route' (212)out: OUTPUT_RULES / REPLY_RULES9Open the source: agent_harness/evals/plan.py:200_as_rule(placed, alert)agent_harness/evals/plan.py:200in: one reply evaluatorout: a Rule named after the evaluator10Open the source: agent_harness/evals/plan.py:148Plan.at(position, alert=None)agent_harness/evals/plan.py:148in: PRE_MODEL / PRE_TOOL / POST_TOOL (246)out: the rules placed at that position11Open the source: agent_harness/evals/screens.py:124rules_at(position, pre_model, post_tool, alert)agent_harness/evals/screens.py:124in: the position and placed evaluatorsout: judging rules, or the hold_writes rule12Open the source: agent_harness/policy/__init__.py:47use_default_rules(DEFAULT_RULES)agent_harness/policy/__init__.py:47in: the table built at line 246out: held as the default table (line 256)13Open the source: agent_harness/policy/__init__.py:63enforce(ctx, rules=None)agent_harness/policy/__init__.py:63in: a Context at run time, no rules givenout: the first blocking Verdict, or ALLOW14Open the source: agent_harness/policy/__init__.py:53default_rules()agent_harness/policy/__init__.py:53in: nothingout: the DEFAULT_RULES table
Read top to bottom: each box is one function call, indented under the function that called it; the dark box is where it starts. Click a box to open its source file.
Dashed boxes are protocols (the ports) or enums; grey boxes are data models; white boxes are classes with behaviour. A dashed arrow means "implements". Click a class to open its source.

How it is done: the real code

Cut from the live repositories each time the site is built, so it always matches what runs.

  P-NO-PROMISE:
    rule: a reply never says or implies that a claim is covered, will be approved or paid, or for what amount, unless the claim's own status and approved amount say so
    via: [get_claim]
    concern: safety
    facet: operational constraint
def no_promise_of_cover(ctx: Context) -> Verdict:
    """P-NO-PROMISE, R-COVERAGE at the output: never say or imply a claim is
    covered, will be approved or paid, or for what amount. A sentence explaining
    that only the assessment decides ("I cannot say whether it is covered") is
    not a promise and is let through (F-004)."""
    for sentence in re.split(r"(?<=[.!?])\s+", ctx.text):
        if PROMISE.search(sentence) and not _EXPLAINING.search(sentence):
            return block("no_promise_of_cover", f"promised an outcome: {sentence[:80]!r}")
    return ALLOW
claims_fnol/policy/__init__.py:190Step 3: registered, with what each rule needs
RULES: dict[str, RuleSpec] = {
    "no_unclaimed_effect": RuleSpec(no_unclaimed_effect, GROUNDED),
    "no_promise_of_cover": RuleSpec(no_promise_of_cover),
    "no_ungrounded_entity": RuleSpec(no_ungrounded_entity, GROUNDED),
    "no_superseded_state": RuleSpec(no_superseded_state, GROUNDED),
    "no_superseded_claim_state": RuleSpec(no_superseded_claim_state, GROUNDED),
    "no_pii_echo": RuleSpec(no_pii_echo),
    "money_in_rupees": RuleSpec(money_in_rupees),
}
claims_fnol/policy/evaluators.yaml:45Step 4: placed at the reply point, with what a fail does
  reply:
    - {use: no_unclaimed_effect, on_fail: safe_reply}
    - {use: no_promise_of_cover, on_fail: safe_reply}
    - {use: no_ungrounded_entity, on_fail: safe_reply}
    - {use: no_superseded_state, on_fail: safe_reply}
    - {use: no_superseded_claim_state, on_fail: safe_reply}
    - {use: no_pii_echo, on_fail: safe_reply}
    - {use: money_in_rupees, on_fail: safe_reply}
agent_harness/policy/__init__.py:63Step 5: enforced; the first block wins, a crash blocks
def enforce(ctx: Context, rules: Sequence[Rule] | None = None) -> Verdict:
    """Run the rules for this position. First block wins.

    A rule that raises **blocks**. That is the whole of AAC-0091: a guardrail
    that errors open is believed and absent at once, and nobody goes looking for
    a control they think they have.
    """
    applicable = default_rules().get(ctx.position, ()) if rules is None else rules
    with tel.span("agent.policy", **{"agent.policy.position": ctx.position.value}) as span:
        for rule in applicable:
            name = getattr(rule, "__name__", repr(rule))
            try:
                verdict = rule(ctx)
            except Exception as exc:  # noqa: BLE001 — deliberate: fail closed
                span.set_attribute("agent.policy.blocked_by", name)
                span.set_attribute("agent.policy.errored", True)
                return block(name, f"rule failed and traffic was blocked: {exc}")
            if verdict.blocked:
                span.set_attribute("agent.policy.blocked_by", verdict.rule or name)
                return verdict
        return ALLOW
tests/test_policy.py:152Step 7: a test that names the policy it proves
@pytest.mark.discharges("P-NO-PROMISE", "AHC-0094")
@pytest.mark.parametrize(
    ("why", "reply", "results", "blocked"), GROUNDING, ids=[g[0] for g in GROUNDING]
)
def test_every_reference_and_figure_is_grounded(
    why: str, reply: str, results: tuple[ToolResult, ...], blocked: bool
) -> None:
    assert no_ungrounded_entity(said(reply, *results)).blocked is blocked


SUPERSEDED = [
    (
clean-ai-engineering/drafts/examples/motor-claims-fnol.aoas.yaml:368The authority-shaped policy: P-PAYOUT's limit, in the spec
    authority:
      agent_when:
        # ASSUMPTION: approved glass claims and small amounts are paid at once;
        # anything larger needs a claims handler. ₹25,000 is a placeholder until
        # Q-AUTO-LIMIT's measurement sets it.
        - {field: approved_amount, at_most: 25000}
      otherwise: human_approval
claims_system/server.py:187And checked again at the far end before any money moves
def covers(
    approval: ApprovalRecord | None,
    *,
    holder: str,
    claim: Row,
    key: str | None,
    now: float,
    limit: Decimal = AOAS_LIMIT,
) -> str | None:
    """Why this approval record does not cover paying this claim now, or `None`.

    The call is rebuilt as this system received it — the claim, the amount this
    system holds for it, the policyholder it verified, the key on the call — and
    its digest must be the record's (`agent_harness.contracts.records.refusals`).
    `limit` is the automatic limit read for this payout."""
    if approval is None:
        return "issue_payout needs an approval, and none this system can read was named"
    amount = Decimal(str(claim.get("approved_amount", 0)))
    failed = refusals(
        approval,
        action="issue_payout",
        args={"claim_id": str(claim["id"]), "amount": str(amount)},
        requested_for=holder,
        idempotency_key=key or "",
        now=now,
    )
    if approval.decided_by in (None, holder):
        failed.append("nobody but the policyholder approved it")
    if amount > limit and approval.decided_by == AUTOMATIC_APPROVER:
        failed.append(f"{amount} is above the {limit} automatic limit, and no person decided it")
    if not failed:
        return None
    return f"approval {approval.id} does not cover this payout: {'; '.join(failed)}"