Eval Harness Tutorial · Architecture
Inside the engine: ports and adapters
A port is a fixed shape: “give me spans”, “judge this request”, “store this result”. The engine only ever talks to ports. Each real tool sits behind an adapter that fits the port, chosen by name in YAML. That's why swapping Phoenix for App Insights, or Presidio for Azure Content Safety, never touches a rule or a report.
The engine's five steps are drawn on the whole picture; this page lists the ports those steps talk through.
| Port | Side | What it does | State |
|---|---|---|---|
TraceSource | Upstream | Reads traces or datasets; yields spans to rebuild sessions from | Built OTLP file built |
InlineHook | Host | Lets a framework, gateway or CI step call our checks at a moment | Not built yet Tier 7 |
Evaluator | Called | One check: request in, result out | Built 8 checks |
Judge | Called | One judge-model setting, handed to each SDK in its own shape | Built |
RuleRunner | Ours | Runs rules over many records (SQL), reused from llm-cost-explainer | Not built yet Tier 5 |
StateStore | Ours | Running totals: budgets, rates, repeat calls | Not built yet Tier 3 |
Policy | Ours | Turns results into one action: allow, redact, block, escalate, alert | Not built yet Tier 3 |
ResultSink | Downstream | Writes results into the team's tools | Not built yet Tier 4 |
AlertSink | Downstream | Sends alerts when a rule over results fires | Not built yet Tier 8 |
How a check's adapter is built, and why it has three parts, is on Design.