cleandataengineering.aiAgent Harness, drawn simply
You are here: End to end › Eval › Rule library

Agent Harness Tutorial · End to end · Zoom level 2

The rule library

Every check, whatever its kind, plugs into the same small shape: a request goes in (the question, the reply, the tools, what was expected), a result comes out (a verdict, a score and a reason). So a check can move between inline, online and offline without being rewritten.

Exact rulecan blockWorld checkreads the system after; can blockLLM judgereports until calibratedHuman labelthe truth judges learn fromevaluate(request) → resultpass · fail · skip · error, score, reason
Solid boxes may block. Dashed boxes may only report.

The library catalogues 92 failure patterns (56 agent failures plus 36 LLM-cost patterns), such as "tool-error ping-pong": the same tool failing twice in one piece of work. The claims agent uses 24 of them as rules today.

For things a rule can't judge, such as "is this reply grounded in the tool results?", the plan is a small self-hosted model rather than a big paid judge. Not built yet.

How it is done: the real code

Cut from the live repositories each time the site is built, so it always matches what runs.

eval-harness-framework/src/evalharness/core.py:238What every check is given (shared shape, from eval-harness-framework)
@dataclass(frozen=True)
class EvalRequest:
    """What every evaluator is handed. Conversation and tools in OpenAI's
    message and tool formats, so a vendor adapter maps fields, not shapes."""

    query: str | None = None
    messages: tuple[Mapping[str, Any], ...] | None = None
    response: Response | None = None
    tool_definitions: tuple[Mapping[str, Any], ...] | None = None
    """The tools this customer was allowed, as OpenAI function definitions."""
    context: tuple[Passage, ...] | None = None
    expected: Expected | None = None
    session: Session | None = None
    """The whole conversation, where a check judges more than one turn."""
    meta: Meta = field(default_factory=Meta)

    def provides(self) -> frozenset[str]:
        """The inputs present on this request, in `NEEDS`' words."""
        response = self.response
        present = {
            "query": self.query is not None,
            "messages": self.messages is not None,
            "response": response is not None,
            "tool_calls": response is not None and response.tool_calls is not None,
            "tool_results": response is not None and response.tool_results is not None,
            "tool_definitions": self.tool_definitions is not None,
            "context": self.context is not None,
            "expected": self.expected is not None,
            "session": self.session is not None,
        }
        return frozenset(name for name, here in present.items() if here)
@dataclass(frozen=True)
class EvalResult:
    """What every evaluator answers, whichever provider it is."""

    verdict: Outcome
    score: float | None
    """Normalised to 0–1, 1 best. `None` when skipped or errored."""
    raw_score: float | None
    """The provider's own number (a 1–5 rating, a 0/1), before normalising."""
    threshold: float | None
    """What `score` was judged against: pass when `score >= threshold`."""
    reason: str
    evaluator: str
    """The configured name: which plug-in this is in `evaluators.yaml`."""
    version: str
    provider: str
    """`ours`, `azure`, an open model's name: whose judgement this is."""
    cost: float = 0.0
    """What the call cost, in USD (tokens or a fee); feeds the cost engine."""
    latency_ms: float = 0.0
    label: str = ""
    """The provider's own label for the outcome (Azure returns one beside its
    score); for a rule, the id its verdict names."""
    raw: Mapping[str, Any] = field(default_factory=dict)
    """The provider's answer, untouched: for audit, and to re-read when the
    mapping changes."""
    findings: tuple[Finding, ...] = ()
    dropped: tuple[str, ...] = ()
    """What the request lost on its way into the provider's shape (passage ids,
    tool definitions), so a lossy result is labelled, not trusted silently."""

    @property
    def failed(self) -> bool:
        return self.verdict == "fail"