Eval Harness Tutorial · Architecture
Who may change a rule, and when it may block
Your decision T22: an agent team may switch off its own rules, except rules the risk owner locks. That needs three layers of rules files, read in order.
Tried when this page was built
| The agent team's file tries to… | Result |
|---|---|
Remove the locked pii rule | refused: pii is locked by the organisation's policy; only its owner (risk-office) can change it |
Switch the locked pii rule to shadow | refused: pii is locked by the organisation's policy; only its owner (risk-office) can change it |
| Raise the unlocked refund cap to ₹20,000 | accepted |
| Remove the unlocked refund cap | accepted |
The audit list
Every change a team's file makes to a rule it inherited is listed, so a reviewer reads one list instead of comparing files:
| Rule | What changed | Before | After |
|---|---|---|---|
refund_cap | max | 10000 | 20000 |
lookup_first | (added) | — | {'adapter': 'requires_before', 'action': 'issue_refund', 'before': 'get_order'} |
The promotion gate: when a rule may block
A rule in shadow records what it would have done. It may move to block only with a named owner and evidence from shadow: precision of at least 0.9 on at least 50 conversations, with the date it was measured. A rule the organisation locked needs only its owner: that's the risk owner's call.
| The file says | Result |
|---|---|
mode: block, nothing else | refused: refund_cap can't block without an owner: add `owner:` (who answers for it) |
| …with an owner, no evidence | refused: refund_cap can't block yet: give its evidence from shadow (`evidence: {precision, sample, measured}`); until then use mode: shadow |
| …with evidence: precision 0.80 on 200 | refused: refund_cap can't block: precision 0.8 on 200 conversations; blocking needs at least 0.9 on at least 50 |
| …with evidence: precision 0.96 on 140 | accepted |
refund_cap:
adapter: argument_limit
tool: issue_refund
field: amount
max: 10000
at: pre_tool
mode: block
owner: claims-ops # who answers for it
evidence: {precision: 0.96, sample: 140, measured: 2026-10-19}The evidence is declared, and checked against the bar. Measuring it for you (a shadow rule's decisions against human labels) comes with the review queue in Tier 8.
- Is 0.9 precision on 50 conversations the right bar to block, or should it differ per kind of rule (a PII mask vs. stopping a refund)?
Reply in chat with the page name and the number, for example “session 2: …”.