cleandataengineering.aiEval Harness, drawn simply
You are here: Architecture › Hooks in your own loop

Eval Harness Tutorial · Architecture

Hooks in a loop of your own

Your question: a framework gives us hooks before and after model and tool calls. In our own loop, code added “before a tool call” runs for every tool. How do we run something before one tool only, and keep injection points simple?

The answer is two separate jobs.

  1. The loop only reports the moment. Five lines, one per moment, each saying “here's what's happening”: which tool, its arguments, the conversation so far. These lines never change, however many rules you add.
  2. Each hook says what it's for. Targeting lives on the hook, as a matcher: one tool by name, a family by pattern (issue_*), or a whole kind of tool (every irreversible one). The loop never has an if tool == ….
Your loop (5 lines added)hooks.run(pre_model) model callhooks.run(post_model)hooks.run(pre_tool) tool runshooks.run(post_tool)hooks.run(reply)eventHooks.run(event)keep the hooks whosemoment and matcher fit,run them in order;first block winsHOOKS, each with a matcherrefund_needs_lookuppre_tool · tools: issue_refundrefund_cappre_tool · tools: issue_refundevery irreversible toolpre_tool · side_effect: irreversiblepii_in_resultspost_tool · every toolredact the replyreply
For an issue_refund call, the first three hooks fit and run; for get_order, none of them do. Adding a rule never touches the loop.

The five lines in your loop

Your own loop, with the five moments marked
for step in range(max_steps):
    hooks.run(Event("pre_model", text=prompt))              # 1
    reply = model(messages)
    hooks.run(Event("post_model", text=reply.text))         # 2
    for call in reply.tool_calls:
        d = hooks.run(Event("pre_tool", tool=call.name,     # 3
                            arguments=call.arguments,
                            side_effect=registry[call.name].side_effect,
                            session=session))
        if d.stops:                      # block or escalate: the tool never runs;
            result = refused(d)          # the model is told why
        else:
            result = tools[call.name](**(d.arguments or call.arguments))
            hooks.run(Event("post_tool", tool=call.name, result=result))   # 4
    ...
hooks.run(Event("reply", text=final_text))                  # 5

Targeting one tool, three ways

In code: decorators, like a framework's hooks
hooks = Hooks()

@hooks.before_tool("issue_refund")                 # one tool
def refund_cap(event):
    if event.arguments["amount"] > 10_000:
        return Decision("escalate", "refunds over ₹10,000 need a person")

@hooks.before_tool("issue_*")                      # a family of tools
def ...

@hooks.before_tool(side_effect="irreversible")     # every tool of a kind
def ...

hooks.add("pre_tool", check(refund_needs_lookup), tools=["issue_refund"])   # any of our checks
In YAML: no code at all (what an ops person writes)
evaluators:
  refund_needs_lookup:
    adapter: requires_before
    action: issue_refund
    before: get_order
    at: pre_tool               # the moment
    tools: [issue_refund]      # the matcher: which tools
    on_fail: block             # block · escalate · allow (just record)

Try it: five tool calls, four hooks

Run when this page was built: the hooks in the drawing, and five tool calls the model asked for. Which hooks ran, and what was decided:

Tool callHooks that ranDecision
get_order, a readnoneallow
issue_refund ₹500, order looked uprefund_needs_lookup, refund_cap, irreversible_needs_named_orderallow
issue_refund ₹500, no lookuprefund_needs_lookupblock: 0 of 1 held; issue_refund without an earlier get_order for the same order_id
issue_refund ₹48,000, looked uprefund_needs_lookup, refund_capescalate: over ₹10,000
cancel_order, irreversibleirreversible_needs_named_orderallow

What a decision does at each moment

Momentallowmodifyblockescalate
before the modelgoes onchanged prompt (redacted)turn ends with a safe replyhand to a person
after the modelgoes onchanged textthe model's answer is not usedhand to a person
before a tooltool runschanged argumentstool doesn't run; the model is told why and may choose againa person approves first
after a toolresult goes into contextredacted resultresult replaced by a notice (the action already happened)hand to a person
the replysentredacted replysafe reply insteadhand to a person

What we borrowed, and from where

IdeaBorrowed fromWhat it gives us
An event plus a matcher on the tool nameClaude Code's PreToolUse and PostToolUse hooksOne tool, a pattern, or all, without touching the loop
A hook answers allow, deny or a changed inputSame hooksHooks can fix a call, not only stop it
A check beside the call that can trip itGuardrails in the agent SDKsOur checks become hooks with one line: check(...)
Before and after the model call, at the gatewayLiteLLM's pre- and post-call guardrailsThe same hook can run at the gateway for model calls (Tier 7)
Matching on the kind of tool, not its nameOur specs: every operation declares its side effectNew irreversible tools are covered without a new rule
Why this stays simple and effective

Simple: one entry point, five calls, one decision type; rules are added in YAML or with a decorator, never in the loop. Effective: unmatched hooks cost nothing; the first block stops the rest; matching by kind of tool covers tools nobody listed; and the same hook runs in your loop, in a framework (an adapter turns its hook into an Event), or at the gateway. Still to come in Tier 3: a time budget per moment, fail-closed when a hook breaks, and every decision recorded on the trace.

Safe by default: modes, broken checks, time budgets

A check that can stop a reply can also stop a good one. So a rule placed in YAML starts in shadow: it runs on every turn and records what it would have done, and changes nothing until someone moves it to block.

ModeRuns?Acts?Use it for
offnonoa rule kept in the file but switched off
shadowyes, and recordednoevery new rule: measure how often it would be right (the default from YAML)
alertyes, and recordedno, but flagged for an alerta rule you trust to notice, not yet to stop
blockyes, and recordedyesa rule with its precision measured and an owner
Each moment's budget and what happens when a check breaks
moments:
  pre_tool: {budget_ms: 200, on_error: closed}   # before an action: never let a broken check through
  reply:    {budget_ms: 300, on_error: open}     # the reply: a broken check shouldn't silence the agent

evaluators:
  refund_cap:
    adapter: argument_limit
    tool: issue_refund
    field: amount
    max: 10000
    at: pre_tool
    mode: shadow                                # the default; move to block when it has earned it
What can go wrongWhat happens
A check is declared slower than its moment allows (Azure groundedness, about 3 s, at pre_tool with 200 ms)Refused when the rules load: “run it online instead”
A check runs over the budget onceIt still decides, and the record says over budget
A blocking check breaks (an exception)on_error: closed blocks the action (the default); open lets the turn go on. Either way it's recorded
A shadow check breaksNothing changes for the customer; the break is recorded

Try it: one rule in shadow, then in block

ModeWhat the rule decidedWhat was applied
shadowblockallow
blockblockblock

Every run of every hook becomes an event on the turn's trace (evalharness.guard.decision): rule, moment, mode, decided, applied, milliseconds, tool, over budget, broke. No reason text, which could quote the conversation. Counting “decided block, applied allow” for a shadow rule over a week is how you measure it before promoting it.

The hook runner, in real codesrc/evalharness/hooks.py:227
    def run(self, event: Event) -> Decision:
        """Every hook for this moment whose matcher fits, in order. The first
        block or escalate wins; a modification is handed on to the next hook.
        Shadow and alert hooks run and are recorded but change nothing; a
        blocking hook that breaks decides by the moment's `on_error`."""
        changed: Decision | None = None
        moment = self.moment(event.position)
        for placed in self.placed:
            if placed.mode == "off":
                continue
            if placed.position != event.position or not placed.match.matches(event):
                continue
            started = time.perf_counter()
            broke = False
            try:
                decision = placed.hook(event) or ALLOW
            except Exception as failure:  # noqa: BLE001 (a hook must never break the turn)
                broke = True
                fallback: Action = "block" if moment.on_error == "closed" else "allow"
                decision = Decision(fallback, reason=f"hook broke: {type(failure).__name__}")
            ms = (time.perf_counter() - started) * 1000
            decision = replace(decision, hook=decision.hook or placed.name)
            acts = placed.mode == "block"
            if self.on_record is not None:
                self.on_record(
                    Record(
                        position=event.position,
                        hook=placed.name,
                        mode=placed.mode,
                        intended=decision.action,
                        applied=decision.action if acts else "allow",
                        ms=ms,
                        tool=event.tool,
                        over_budget=moment.budget_ms is not None and ms > moment.budget_ms,
                        broke=broke,
                    )
                )
            if not acts:
                continue
            if decision.stops:
                return decision
            if decision.action == "modify":
                changed = decision
                event = replace(
                    event,
                    arguments=decision.arguments
                    if decision.arguments is not None
                    else event.arguments,
                    text=decision.text if decision.text is not None else event.text,
                )
        return changed or ALLOW


class CheckBroke(RuntimeError):
    """A check answered `error`: it tried and broke. Raised so the moment's
    `on_error` decides, as for any hook that breaks."""


@runtime_checkable
class Policy(Protocol):
    """The decide step: one check's result becomes one decision. Kept apart from
    the check, so the same check can block in one company and redact in another."""

    def decide(self, result: EvalResult, event: Event, on_fail: Action) -> Decision | None: ...


class OnFail:
    """A fail does what the rule says (`on_fail`); anything else allows."""

    def decide(self, result: EvalResult, event: Event, on_fail: Action) -> Decision | None:
        if result.verdict == "fail":
            return Decision(on_fail, reason=result.reason, hook=result.evaluator)
        return None


class Redact:
    """For text the check located (Presidio gives each finding's start and end):
    mask those parts and let the text through, instead of blocking it. A failing
    result with nothing located falls back to `on_fail`."""

    def __init__(self, mask: str = "[{kind}]") -> None:
        self.mask = mask

    def decide(self, result: EvalResult, event: Event, on_fail: Action) -> Decision | None:
        if result.verdict != "fail":
            return None
        spans = merged(
            (f.start, f.end, f.kind)
            for f in result.findings
            if f.start is not None and f.end is not None
        )
        if not spans or not event.text:
            return OnFail().decide(result, event, on_fail)
        text, cut, last = event.text, [], 0
        for start, end, kind in spans:
            cut.append(text[last:start] + self.mask.format(kind=kind))
            last = end
        cut.append(text[last:])
        return Decision(
            "modify", reason=f"{len(spans)} masked", hook=result.evaluator, text="".join(cut)
        )


def merged(spans: Any) -> list[tuple[int, int, str]]:
    """Overlapping hits as one region each, named by its widest hit: Presidio
    finds "URLs" inside an email address, and masking only the first hit would
    leave the rest of the address showing."""
    out: list[tuple[int, int, str, int]] = []
    for start, end, kind in sorted(spans):
        if out and start < out[-1][1]:
            s0, e0, k0, w0 = out[-1]
            width = end - start
            out[-1] = (s0, max(e0, end), kind if width > w0 else k0, max(w0, width))
        else:
            out.append((start, end, kind, end - start))
    return [(s0, e0, k0) for s0, e0, k0, _ in out]
Decided · Hooks in your own loop
  1. Is “the loop reports the moment, the hook says what it's for” simple enough to explain to an agent team?
    Decided: Recommended: yes, and lead with matching by kind of tool (irreversible, write), because it covers new tools without new rules.

Where each moment sits in a turn: offline, inline, online.