Agent Harness Tutorial · End to end · Zoom level 3
How we take care of it: no_unclaimed_effect
no_unclaimed_effect catches "I've registered it" when nothing was registered.
How we took care of it.
- If the reply says something was done (registered, withdrawn, paid, sent), look for a successful write tool call this turn that did it.
- No such call means the reply is claiming an effect that never happened.
| It reads | the reply, and this turn's tool calls and results |
| Kind | rule |
| Provided by | agent |
| If it fails | safe reply |
| Where in the code | F policy rules · evaluators.yaml reply |
| Runs at | 3 · After the model |
Real exampleThe case that started our eval work: "Yes, your claim is being processed" when no claim was created.
How it is done: the real code
Cut from the live repositories each time the site is built, so it always matches what runs.
def no_unclaimed_effect(ctx: Context) -> Verdict:
"""An effect may be *claimed* only if it happened this turn."""
for action, pattern in CLAIM_PATTERNS.items():
if pattern.search(ctx.text) and not any(_succeeded(r, action) for r in ctx.tool_results):
return block("no_unclaimed_effect", f"claimed {action} happened when it did not")
return ALLOW