Agent Harness Tutorial · End to end · Zoom level 2
The rule library
Every check, whatever its kind, plugs into the same small shape: a request goes in (the question, the reply, the tools, what was expected), a result comes out (a verdict, a score and a reason). So a check can move between inline, online and offline without being rewritten.
The library catalogues 92 failure patterns (56 agent failures plus 36 LLM-cost patterns), such as "tool-error ping-pong": the same tool failing twice in one piece of work. The claims agent uses 24 of them as rules today.
For things a rule can't judge, such as "is this reply grounded in the tool results?", the plan is a small self-hosted model rather than a big paid judge. Not built yet.
How it is done: the real code
Cut from the live repositories each time the site is built, so it always matches what runs.
eval-harness-framework/src/evalharness/core.py:238What every check is given (shared shape, from eval-harness-framework)@dataclass(frozen=True)
class EvalRequest:
"""What every evaluator is handed. Conversation and tools in OpenAI's
message and tool formats, so a vendor adapter maps fields, not shapes."""
query: str | None = None
messages: tuple[Mapping[str, Any], ...] | None = None
response: Response | None = None
tool_definitions: tuple[Mapping[str, Any], ...] | None = None
"""The tools this customer was allowed, as OpenAI function definitions."""
context: tuple[Passage, ...] | None = None
expected: Expected | None = None
session: Session | None = None
"""The whole conversation, where a check judges more than one turn."""
meta: Meta = field(default_factory=Meta)
def provides(self) -> frozenset[str]:
"""The inputs present on this request, in `NEEDS`' words."""
response = self.response
present = {
"query": self.query is not None,
"messages": self.messages is not None,
"response": response is not None,
"tool_calls": response is not None and response.tool_calls is not None,
"tool_results": response is not None and response.tool_results is not None,
"tool_definitions": self.tool_definitions is not None,
"context": self.context is not None,
"expected": self.expected is not None,
"session": self.session is not None,
}
return frozenset(name for name, here in present.items() if here)eval-harness-framework/src/evalharness/core.py:313What every check gives back@dataclass(frozen=True)
class EvalResult:
"""What every evaluator answers, whichever provider it is."""
verdict: Outcome
score: float | None
"""Normalised to 0–1, 1 best. `None` when skipped or errored."""
raw_score: float | None
"""The provider's own number (a 1–5 rating, a 0/1), before normalising."""
threshold: float | None
"""What `score` was judged against: pass when `score >= threshold`."""
reason: str
evaluator: str
"""The configured name: which plug-in this is in `evaluators.yaml`."""
version: str
provider: str
"""`ours`, `azure`, an open model's name: whose judgement this is."""
cost: float = 0.0
"""What the call cost, in USD (tokens or a fee); feeds the cost engine."""
latency_ms: float = 0.0
label: str = ""
"""The provider's own label for the outcome (Azure returns one beside its
score); for a rule, the id its verdict names."""
raw: Mapping[str, Any] = field(default_factory=dict)
"""The provider's answer, untouched: for audit, and to re-read when the
mapping changes."""
findings: tuple[Finding, ...] = ()
dropped: tuple[str, ...] = ()
"""What the request lost on its way into the provider's shape (passage ids,
tool definitions), so a lossy result is labelled, not trusted silently."""
@property
def failed(self) -> bool:
return self.verdict == "fail"