Agent Harness Tutorial · End to end · Zoom level 3
How we take care of it: no_ungrounded_entity
no_ungrounded_entity catches a claim number no tool returned.
How we took care of it.
- Find every CLM- and POL- number in the reply.
- Each must have come back from a tool this turn, or from a fresh re-read.
| It reads | the reply, and the ids that tools returned this turn |
| Kind | rule |
| Provided by | agent |
| If it fails | safe reply |
| Where in the code | F policy rules · evaluators.yaml reply |
| Runs at | 3 · After the model |
Real exampleStops a made-up claim number from ever reaching Rohan.
How it is done: the real code
Cut from the live repositories each time the site is built, so it always matches what runs.
def no_ungrounded_entity(ctx: Context) -> Verdict:
"""Every reference, amount and date in the reply appears in what the tools
returned this turn — so no figure nobody decided reaches the policyholder."""
if not ctx.tool_results:
return ALLOW
evidence = " ".join(f"{r.structured} {r.text}" for r in ctx.tool_results).lower()
for found in set(IDENTIFIER.findall(ctx.text)):
if found.lower() not in evidence:
return block("no_ungrounded_entity", f"cited {found!r}, which no tool returned")
for date in ISO_DATE.findall(ctx.text):
if date not in evidence:
return block("no_ungrounded_entity", f"stated the date {date!r}, unsupported")
flat = _flat(evidence)
for match in MONEY.finditer(IDENTIFIER.sub(" ", ctx.text)):
amount = match.group(1) or match.group(2)
if amount and (_flat(amount).lstrip("0") or "0") not in flat:
return block("no_ungrounded_entity", f"stated the figure {amount!r}, unsupported")
return ALLOW