Eval Harness Tutorial · Design
Every decision taken so far
34 decisions, read from docs/PLAN.html when this page was built,
so it is always the plan's own list. To reopen one, say its number.
| # | Decision | Why | When |
|---|---|---|---|
| T1 | Build it, but only the parts that don't exist yet | Most pieces exist; the glue, the honest mix and the spec link don't. | 17 Oct |
| T2 | IoC: every vendor, cloud and check behind a port; swapping one is an adapter plus a YAML line | The same framework must run open-stack, on Azure, and later on AWS or GCP. | 17 Oct |
| T3 | The Assurance Layer is the spec; this is the implementation | Keeps "what must be true" separate from "how we check it". | 17 Oct |
| T4 | Security and cost checks are in scope; identity, network, secrets and billing are not | Anything that looks at what the agent does belongs here; anything that controls access to it belongs to the platform. | 17 Oct |
| T5 | No database, UI, dashboards or semantic engine of our own; fit into existing tools and ship content for them | They already exist, and teams won't move off theirs. | 17 Oct |
| T6 | The USP, sharpened by the landscape check: checks traced to a spec; deterministic "owed but not done" checks over the whole trace; cost, safety and quality under one verdict that can block. Mixing providers honestly (one scale, declared losses, cost per check) supports it. | Every platform already lets you attach outside scores, so mixing alone isn't new. No product found links checks to obligations; omission checks elsewhere are LLM-judged only; nobody treats cost rules as blocking guardrails. | 17 Oct |
| T7 | The 36 cost rules are covered; llm-cost-explainer stays its own product and is also used as a library | Its rule runner is the population engine we need. | 17 Oct |
| T8 | Public repo, Apache-2.0 | Same as your other public repos. | 17 Oct |
| T9 | Free judge models (Groq) for our own and demo work; for clients, the judge is their own Azure OpenAI or Bedrock | Your free-models rule for personal projects; client traces sent to Groq would fail an enterprise security review. | standing |
| T10 | Package name evalharness (was D1) | One name for repo and package. Free on PyPI as of 17 Oct; to be claimed at publish. | 17 Oct |
| T11 | Extract, don't copy: agent_harness.evals re-exports from evalharness (was D2) | One copy of the classes. Done the same day; reference-agent and FNOL suites unchanged. | 17 Oct |
| T12 | Phoenix as the local backend (was D4) | One Python process; self-hosted Langfuse needs four services. | 17 Oct |
| T13 | Build version 1 (the simple slice) first (was D13) | Tests whether teams find real problems before we build inline, alerts and integrations. | 17 Oct |
| T14 | Fix FNOL's trace gap in the harness (was D14): approvals' work carries the conversation's id; local tools record what they did | Done the same day. All five patterns now pass on all 29 FNOL scenarios with nothing left to explain. | 18 Oct |
| T15 | Inline checks run first in framework hooks; LiteLLM second, for non-Python agents (was D3) | Only framework hooks see tool calls. Accepted with the other recommendations on 18 Oct. | 18 Oct |
| T16 | Metric definitions first for Cube (open track) and Databricks Metric Views (Databricks clients) (was D5) | Follows the earlier "by track" choice. Accepted with the other recommendations on 18 Oct. | 18 Oct |
| T17 | Online checks read traces from where they land (pull) first; a Collector copy (tap) second (was D6) | Works with existing projects and allows backfill. Accepted with the other recommendations on 18 Oct. | 18 Oct |
| T18 | The conversation layer is one small MCP server over the client's semantic layer; no UI of our own (was D7) | Any chat surface can use it. Accepted with the other recommendations on 18 Oct. | 18 Oct |
| T19 | No checking of replies while they stream, for now (was D8) | FNOL and clothing don't stream to the customer; revisit when an agent does. Accepted with the other recommendations on 18 Oct. | 18 Oct |
| T20 | Carry the work after results land, scoped: jobs once against ports, a five-method adapter per tool, a short supported list, steps 1–6 first (was D9) | Keeps the loop's work bounded. Accepted with the other recommendations on 18 Oct. | 18 Oct |
| T21 | Trial two cost rules as a dbt package before choosing between dbt and our own runner (was D10) | Decide on evidence. Accepted with the other recommendations on 18 Oct. | 18 Oct |
| T22 | The agent team may switch its own rules off, except rules the risk owner locks (was D11) | Teams keep freedom; the org keeps its minimum. Accepted with the other recommendations on 18 Oct. | 18 Oct |
| T23 | Results published as OTel's gen_ai.evaluation.result, extended with scale, threshold, findings and rule identity (was D12) | Any tool can read them without our code. Accepted with the other recommendations on 18 Oct. | 18 Oct |
| T24 | Session: keep five kinds of step; a guardrail block is a blocked decision; a person's reply is an assistant step named human; every step has an optional time; outcomes are resolved, handed_over, abandoned, error | Covers what ops asks without new kinds. From the tutorial review. | 18 Oct |
| T25 | Patterns: phrase matching first, AI judge later as an opt-in per rule; YAML keeps under/over, reports say missed/needless; consistency guards amounts and ids now, dates next | Same verdict every run until a team chooses to pay for a judge. From the tutorial review. | 18 Oct |
| T26 | Every rule takes a plain description; ready-made fact types replace hand-written regular expressions | An ops person should be able to write and read the rules file. From the tutorial review. | 18 Oct |
| T27 | No hand-over proposal to OpenTelemetry until a second team needs it; meanwhile a team maps its own attribute in YAML | One team's need is not yet a standard. From the tutorial review. | 18 Oct |
| T28 | Trace sources in this order: OTLP file (built), Phoenix, App Insights, Langfuse | App Insights is where the Azure clients and FNOL already are. From the tutorial review. | 18 Oct |
| T29 | Inline hooks: the loop only reports the moment (five calls to Hooks.run); each hook carries a matcher (tool name, pattern such as issue_*, or kind of tool such as irreversible); one decision type (allow, modify, block, escalate); first block wins. Placed by decorator or in YAML (at, tools, on_fail). Framework and gateway adapters only translate their hook into an Event. | Borrowed from Claude Code's PreToolUse/PostToolUse matchers, the agent SDKs' guardrails and LiteLLM's pre/post-call guardrails. Targeting one tool never touches the loop; matching by kind covers new tools. Your question, 19 Oct. | 19 Oct |
| T30 | Checks that need the conversation's words run inside the agent's environment; only verdicts go to App Insights, content-free by default (counts and finding kinds, never quoted words or amounts). Content capture stays off in production; a dev deployment with synthetic customers may turn it on. | Production traces with privacy on show what happened, not what was said (run 4b). Moving words into a trace store would fail a regulated client's review. Your choice, 19 Oct. | 19 Oct |
| T31 | A stack binding names stores by role (detail, numbers, history) and says what each can do; each ops question is routed to the cheapest role that can answer it, or reported as “can't be answered here” (was D15) | Azure keeps everything in App Insights; the open stack splits Langfuse and Prometheus. Checks and questions must not change between them. Your choice, 16 Nov. | 16 Nov |
| T32 | Counters are added up by the Collector where there is one, by the store at question time when it is a single store that keeps every span, and never from sampled traces (was D16) | A total counted from a sample is wrong, and nobody can see that it is. | 16 Nov |
| T33 | Every verdict is written as an event on the trace and as a counter, from one call (was D17) | The event opens the conversation; the counter makes rates and alerts cheap and survives sampling. | 16 Nov |
| T34 | On Azure the default is App Insights alone; managed Prometheus is added when alerts must run more often than every 15 minutes, traces are sampled, or many agents share one board (was D18) | Nothing extra to run until one of the three is true. | 16 Nov |