cleandataengineering.aiEval Harness, drawn simply
You are here: Architecture › Decide and count

Eval Harness Tutorial · Architecture

Deciding, masking, and counting across conversations

A check says what it found. The decide step (the Policy port) turns that into one action, so the same check can block in one company and mask in another.

PolicyWhat a fail becomes
OnFail (the default)the rule's on_fail: block, escalate, or allow and record
Redactfor text the check located (Presidio gives each finding's position): the text goes through with those parts masked

Masking a reply with Presidio

Text
The agent wroteYour refund is on its way to priya.sharma@example.com; call 9876543210 if it isn't.
The customer sawYour refund is on its way to [EMAIL_ADDRESS]; call [PHONE_NUMBER] if it isn't.

run with Presidio when this page was built.

A bug the real run caught

Presidio reports overlapping hits: an email address, and “URLs” inside it. Masking the first hit only produced [URL]arma@[URL], leaking part of the address. Overlapping hits are now merged into one region, named by the widest hit.

A check that breaks never lets the action through

Our adapters catch a vendor's exception and answer error. Read as “no problem”, that let actions through when a check broke: the opposite of fail-closed. An error now counts as a broken hook, so the moment's on_error decides (closed by default, see hooks).

Limits across conversations

“Refunds to one customer in 30 days must stay under ₹20,000” can't be judged in one conversation. A running total per customer (the StateStore port) answers it: before issue_refund runs, would this refund take customer C-17's 30-day total over the limit? Three refunds, run when this page was built:

Refund askedDecisionWhy
₹12,000allow—
₹6,000allow—
₹4,000escalate22000 in the window would pass 20000

Only refunds that succeeded are counted. The in-memory store suits tests and one worker; several workers need a shared store (Redis), which is the same port.

The masking policy, in real codesrc/evalharness/hooks.py:302
class Redact:
    """For text the check located (Presidio gives each finding's start and end):
    mask those parts and let the text through, instead of blocking it. A failing
    result with nothing located falls back to `on_fail`."""

    def __init__(self, mask: str = "[{kind}]") -> None:
        self.mask = mask

    def decide(self, result: EvalResult, event: Event, on_fail: Action) -> Decision | None:
        if result.verdict != "fail":
            return None
        spans = merged(
            (f.start, f.end, f.kind)
            for f in result.findings
            if f.start is not None and f.end is not None
        )
        if not spans or not event.text:
            return OnFail().decide(result, event, on_fail)
        text, cut, last = event.text, [], 0
        for start, end, kind in spans:
            cut.append(text[last:start] + self.mask.format(kind=kind))
            last = end
        cut.append(text[last:])
        return Decision(
            "modify", reason=f"{len(spans)} masked", hook=result.evaluator, text="".join(cut)
        )