Eval Harness Tutorial · Design
Design: the ideas the code is built on
Eight ideas run through the code. Each one is here with the reason for it; the decisions page lists every choice taken so far.
| Idea | What it means | Why |
|---|---|---|
| One request shape, one result shape | Every check, ours or a vendor's, is handed the same EvalRequest and answers with the same EvalResult. Reports never care who judged. | The spike ran four providers through one shape and it held. |
| Ports and adapters | The engine talks to fixed shapes (ports); each tool fits one through an adapter chosen in YAML. | The same rules must run on the open stack, Azure, and later AWS or GCP. |
| Three parts per adapter | Translate in, call, translate out. Only the middle part imports the vendor's SDK. | Vendor SDKs change monthly; the change stays in one small function, and recorded answers test the other two. |
| One scale | Every score becomes 0–1 with 1 best, from the provider's own range and threshold set in YAML. | Azure scores 1–5, others 0–1, some lower-is-better. Mixing them honestly needs one scale. |
| Say what was lost | A result lists what didn't fit into the provider's shape (dropped) and keeps the provider's raw answer. | A lossy result must be labelled, not trusted silently. |
| Confidence is not severity | How sure a provider is and how bad a finding is are separate numbers. | Presidio is surer of a date than of a URL, yet an email leak is worse than either. |
| Checking is separate from deciding | A check says what it found; a policy step decides block, alert or count. | The same check can block in one company and only alert in another. |
| Rules are data, linked to requirements | Rules are YAML; each names the requirements it tests and each check the standard's obligations. | Every fail says which promise broke; a test catches a broken link. |
The core classes
The adapter, in real code
The three parts, plus what every check declares
src/evalharness/adapter.py:22class Adapter:
kind: ClassVar[str]
"""The registry name, as `evaluators.yaml` writes it (`adapter: <kind>`)."""
version: ClassVar[str]
"""The provider SDK version this mapping was written and recorded against."""
provider: ClassVar[str]
needs: ClassVar[frozenset[str]]
carries: ClassVar[frozenset[str]]
"""Which inputs the provider's shape has room for."""
wants: ClassVar[frozenset[str] | None] = None
"""What the check would judge on if the provider had room: `carries` when
`None`. Only inputs it wants and cannot carry are reported `dropped`."""
lossy: ClassVar[tuple[str, ...]] = ()
"""What is carried but loses detail on the way (passage ids, call ids)."""
speed: ClassVar[Speed]
expected_ms: ClassVar[float]
default_scale: ClassVar[Scale]
uses_judge: ClassVar[bool] = False
stability: ClassVar[Stability] = "alpha"
obligations: ClassVar[tuple[str, ...]] = ()
"""The AI Assurance Catalog obligations (`AAC-####`) this check helps
discharge, whatever the agent: what links it to the standard."""
def __init__(
self,
*,
name: str | None = None,
judge: JudgeModel | None = None,
scale: Scale | None = None,
description: str = "",
requirements: Sequence[str] = (),
) -> None:
self.name = name or self.kind
self.description = description
"""The rule in plain words, for reports: what an ops reader sees beside the verdict."""
self.governance: dict[str, Any] = {}
"""Who owns it, whether the organisation locked it, its evidence for blocking."""
self.placement: dict[str, Any] = {}
"""Where the file places it inline (`at`, `tools`, `on_fail`); empty when not."""
self.requirements = tuple(requirements)
"""The requirements this rule tests, by id in the team's spec (`P-PAYOUT`,
`escalation.injury-reported`): what links it to what must be true."""
self.judge = judge
self.scale = scale or self.default_scale
# -- the three parts -------------------------------------------------------
def to_provider(self, request: EvalRequest) -> Any:
raise NotImplementedError
def call(self, payload: Any) -> Mapping[str, Any]:
raise NotImplementedError
def from_provider(self, raw: Mapping[str, Any], request: EvalRequest) -> EvalResult:
raise NotImplementedError
# -- shared ----------------------------------------------------------------Where these classes sit in the whole system: inside the engine.