cleandataengineering.aiAgent Harness, drawn simply
You are here: End to end › Checks › 3 · After the model › no_unclaimed_effect

Agent Harness Tutorial · End to end · Zoom level 3

How we take care of it: no_unclaimed_effect

no_unclaimed_effect catches "I've registered it" when nothing was registered.

Open: 11before loopopen ›Open: 22before modelopen ›Open: 33after modelopen ›Open: 44before toolopen ›Open: 55the toolopen ›Open: 66after toolopen ›Open: 77after loopopen ›Open: 88before replyopen ›It readsthe reply, and this turn's toolcalls and resultsno_unclaimed_effectrule · agentpassCarries onfailIf it failssafe replyWHERE IN THE CODE: F policy rules · evaluators.yaml reply
The strip at the top shows every check point where this check runs, in dark.

How we took care of it.

  1. If the reply says something was done (registered, withdrawn, paid, sent), look for a successful write tool call this turn that did it.
  2. No such call means the reply is claiming an effect that never happened.
It readsthe reply, and this turn's tool calls and results
Kindrule
Provided byagent
If it failssafe reply
Where in the codeF policy rules · evaluators.yaml reply
Runs at3 · After the model
Real exampleThe case that started our eval work: "Yes, your claim is being processed" when no claim was created.

More on this: Eval.

How it is done: the real code

Cut from the live repositories each time the site is built, so it always matches what runs.

def no_unclaimed_effect(ctx: Context) -> Verdict:
    """An effect may be *claimed* only if it happened this turn."""
    for action, pattern in CLAIM_PATTERNS.items():
        if pattern.search(ctx.text) and not any(_succeeded(r, action) for r in ctx.tool_results):
            return block("no_unclaimed_effect", f"claimed {action} happened when it did not")
    return ALLOW