cleandataengineering.aiEval Harness, drawn simply
You are here: Architecture › Who may change a rule

Eval Harness Tutorial · Architecture

Who may change a rule, and when it may block

Your decision T22: an agent team may switch off its own rules, except rules the risk owner locks. That needs three layers of rules files, read in order.

1 · Presetready-made checksfor a kind of agent2 · The organisation's policyrisk office's file;can lock rules3 · The agent team's fileadds, changes, removes;can't touch locked rulesRules+ audit list
Each layer is laid over the one before, field by field. A locked rule from the organisation passes through untouched.

Tried when this page was built

The agent team's file tries to…Result
Remove the locked pii rulerefused: pii is locked by the organisation's policy; only its owner (risk-office) can change it
Switch the locked pii rule to shadowrefused: pii is locked by the organisation's policy; only its owner (risk-office) can change it
Raise the unlocked refund cap to ₹20,000accepted
Remove the unlocked refund capaccepted

The audit list

Every change a team's file makes to a rule it inherited is listed, so a reviewer reads one list instead of comparing files:

RuleWhat changedBeforeAfter
refund_capmax1000020000
lookup_first(added)—{'adapter': 'requires_before', 'action': 'issue_refund', 'before': 'get_order'}

The promotion gate: when a rule may block

A rule in shadow records what it would have done. It may move to block only with a named owner and evidence from shadow: precision of at least 0.9 on at least 50 conversations, with the date it was measured. A rule the organisation locked needs only its owner: that's the risk owner's call.

The file saysResult
mode: block, nothing elserefused: refund_cap can't block without an owner: add `owner:` (who answers for it)
…with an owner, no evidencerefused: refund_cap can't block yet: give its evidence from shadow (`evidence: {precision, sample, measured}`); until then use mode: shadow
…with evidence: precision 0.80 on 200refused: refund_cap can't block: precision 0.8 on 200 conversations; blocking needs at least 0.9 on at least 50
…with evidence: precision 0.96 on 140accepted
A rule that has earned the right to block
refund_cap:
  adapter: argument_limit
  tool: issue_refund
  field: amount
  max: 10000
  at: pre_tool
  mode: block
  owner: claims-ops                                # who answers for it
  evidence: {precision: 0.96, sample: 140, measured: 2026-10-19}
Not built yet

The evidence is declared, and checked against the bar. Measuring it for you (a shadow rule's decisions against human labels) comes with the review queue in Tier 8.

Your review · Who may change a rule
  1. Is 0.9 precision on 50 conversations the right bar to block, or should it differ per kind of rule (a PII mask vs. stopping a refund)?

Reply in chat with the page name and the number, for example “session 2: …”.