cleandataengineering.aiAgent Harness, drawn simply
You are here: End to end › Checks › 3 · After the model

Agent Harness Tutorial · End to end · Zoom level 2

Check point 3: After the model

The model has written an answer. Before anyone sees it, check it against what the tools actually said.

Where: when the model answers in words.

The strip at the top is the whole turn; the dark box is where you are. Click a check, or what it catches, to see how we took care of it. Dashed boxes are not built yet.

Most of these rules are grounded: they compare the answer with what the tools actually returned in this turn. "Your claim CLM-123 is approved" fails if no tool returned CLM-123, or returned it with a different status.

How it is done: the real code

Cut from the live repositories each time the site is built, so it always matches what runs.

claims_fnol/policy/evaluators.yaml:45evaluators.yaml: the reply rules
  reply:
    - {use: no_unclaimed_effect, on_fail: safe_reply}
    - {use: no_promise_of_cover, on_fail: safe_reply}
    - {use: no_ungrounded_entity, on_fail: safe_reply}
    - {use: no_superseded_state, on_fail: safe_reply}
    - {use: no_superseded_claim_state, on_fail: safe_reply}
    - {use: no_pii_echo, on_fail: safe_reply}
    - {use: money_in_rupees, on_fail: safe_reply}

Look inside