cleandataengineering.aiEval Harness, drawn simply
You are here: Design

Eval Harness Tutorial · Design

Design: the ideas the code is built on

Eight ideas run through the code. Each one is here with the reason for it; the decisions page lists every choice taken so far.

IdeaWhat it meansWhy
One request shape, one result shapeEvery check, ours or a vendor's, is handed the same EvalRequest and answers with the same EvalResult. Reports never care who judged.The spike ran four providers through one shape and it held.
Ports and adaptersThe engine talks to fixed shapes (ports); each tool fits one through an adapter chosen in YAML.The same rules must run on the open stack, Azure, and later AWS or GCP.
Three parts per adapterTranslate in, call, translate out. Only the middle part imports the vendor's SDK.Vendor SDKs change monthly; the change stays in one small function, and recorded answers test the other two.
One scaleEvery score becomes 0–1 with 1 best, from the provider's own range and threshold set in YAML.Azure scores 1–5, others 0–1, some lower-is-better. Mixing them honestly needs one scale.
Say what was lostA result lists what didn't fit into the provider's shape (dropped) and keeps the provider's raw answer.A lossy result must be labelled, not trusted silently.
Confidence is not severityHow sure a provider is and how bad a finding is are separate numbers.Presidio is surer of a date than of a URL, yet an email leak is worse than either.
Checking is separate from decidingA check says what it found; a policy step decides block, alert or count.The same check can block in one company and only alert in another.
Rules are data, linked to requirementsRules are YAML; each names the requirements it tests and each check the standard's obligations.Every fail says which promise broke; a test catches a broken link.

The core classes

EvalRequestquery, messagesresponse (+ tool calls, results)context passages · expectedsession · metaAdapter (a check)to_provider(request)call(payload) ← only part that touches the vendorfrom_provider(raw) → resultobligations · requirementsEvalResultverdict: pass·fail·skip·errorscore 0–1 · raw_score · thresholdreason · cost · latencyraw · findings · droppedinoutOpen: SessionSessionsteps in order(see Sessions)open ›Scalethe provider's min, max,direction, threshold → 0–1Findingkind · confidence · severitywhere in the text
A request goes into an adapter and a result comes out. Every check, ours or a vendor's, has this shape.

The adapter, in real code

The three parts, plus what every check declaressrc/evalharness/adapter.py:22
class Adapter:
    kind: ClassVar[str]
    """The registry name, as `evaluators.yaml` writes it (`adapter: <kind>`)."""
    version: ClassVar[str]
    """The provider SDK version this mapping was written and recorded against."""
    provider: ClassVar[str]
    needs: ClassVar[frozenset[str]]
    carries: ClassVar[frozenset[str]]
    """Which inputs the provider's shape has room for."""
    wants: ClassVar[frozenset[str] | None] = None
    """What the check would judge on if the provider had room: `carries` when
    `None`. Only inputs it wants and cannot carry are reported `dropped`."""
    lossy: ClassVar[tuple[str, ...]] = ()
    """What is carried but loses detail on the way (passage ids, call ids)."""
    speed: ClassVar[Speed]
    expected_ms: ClassVar[float]
    default_scale: ClassVar[Scale]
    uses_judge: ClassVar[bool] = False
    stability: ClassVar[Stability] = "alpha"
    obligations: ClassVar[tuple[str, ...]] = ()
    """The AI Assurance Catalog obligations (`AAC-####`) this check helps
    discharge, whatever the agent: what links it to the standard."""

    def __init__(
        self,
        *,
        name: str | None = None,
        judge: JudgeModel | None = None,
        scale: Scale | None = None,
        description: str = "",
        requirements: Sequence[str] = (),
    ) -> None:
        self.name = name or self.kind
        self.description = description
        """The rule in plain words, for reports: what an ops reader sees beside the verdict."""
        self.governance: dict[str, Any] = {}
        """Who owns it, whether the organisation locked it, its evidence for blocking."""
        self.placement: dict[str, Any] = {}
        """Where the file places it inline (`at`, `tools`, `on_fail`); empty when not."""
        self.requirements = tuple(requirements)
        """The requirements this rule tests, by id in the team's spec (`P-PAYOUT`,
        `escalation.injury-reported`): what links it to what must be true."""
        self.judge = judge
        self.scale = scale or self.default_scale

    # -- the three parts -------------------------------------------------------

    def to_provider(self, request: EvalRequest) -> Any:
        raise NotImplementedError

    def call(self, payload: Any) -> Mapping[str, Any]:
        raise NotImplementedError

    def from_provider(self, raw: Mapping[str, Any], request: EvalRequest) -> EvalResult:
        raise NotImplementedError

    # -- shared ----------------------------------------------------------------

Where these classes sit in the whole system: inside the engine.

Look inside