cleandataengineering.aiEval Harness, drawn simply
You are here: Design › Decisions

Eval Harness Tutorial · Design

Every decision taken so far

34 decisions, read from docs/PLAN.html when this page was built, so it is always the plan's own list. To reopen one, say its number.

#DecisionWhyWhen
T1Build it, but only the parts that don't exist yetMost pieces exist; the glue, the honest mix and the spec link don't.17 Oct
T2IoC: every vendor, cloud and check behind a port; swapping one is an adapter plus a YAML lineThe same framework must run open-stack, on Azure, and later on AWS or GCP.17 Oct
T3The Assurance Layer is the spec; this is the implementationKeeps "what must be true" separate from "how we check it".17 Oct
T4Security and cost checks are in scope; identity, network, secrets and billing are notAnything that looks at what the agent does belongs here; anything that controls access to it belongs to the platform.17 Oct
T5No database, UI, dashboards or semantic engine of our own; fit into existing tools and ship content for themThey already exist, and teams won't move off theirs.17 Oct
T6The USP, sharpened by the landscape check: checks traced to a spec; deterministic "owed but not done" checks over the whole trace; cost, safety and quality under one verdict that can block. Mixing providers honestly (one scale, declared losses, cost per check) supports it.Every platform already lets you attach outside scores, so mixing alone isn't new. No product found links checks to obligations; omission checks elsewhere are LLM-judged only; nobody treats cost rules as blocking guardrails.17 Oct
T7The 36 cost rules are covered; llm-cost-explainer stays its own product and is also used as a libraryIts rule runner is the population engine we need.17 Oct
T8Public repo, Apache-2.0Same as your other public repos.17 Oct
T9Free judge models (Groq) for our own and demo work; for clients, the judge is their own Azure OpenAI or BedrockYour free-models rule for personal projects; client traces sent to Groq would fail an enterprise security review.standing
T10Package name evalharness (was D1)One name for repo and package. Free on PyPI as of 17 Oct; to be claimed at publish.17 Oct
T11Extract, don't copy: agent_harness.evals re-exports from evalharness (was D2)One copy of the classes. Done the same day; reference-agent and FNOL suites unchanged.17 Oct
T12Phoenix as the local backend (was D4)One Python process; self-hosted Langfuse needs four services.17 Oct
T13Build version 1 (the simple slice) first (was D13)Tests whether teams find real problems before we build inline, alerts and integrations.17 Oct
T14Fix FNOL's trace gap in the harness (was D14): approvals' work carries the conversation's id; local tools record what they didDone the same day. All five patterns now pass on all 29 FNOL scenarios with nothing left to explain.18 Oct
T15Inline checks run first in framework hooks; LiteLLM second, for non-Python agents (was D3)Only framework hooks see tool calls. Accepted with the other recommendations on 18 Oct.18 Oct
T16Metric definitions first for Cube (open track) and Databricks Metric Views (Databricks clients) (was D5)Follows the earlier "by track" choice. Accepted with the other recommendations on 18 Oct.18 Oct
T17Online checks read traces from where they land (pull) first; a Collector copy (tap) second (was D6)Works with existing projects and allows backfill. Accepted with the other recommendations on 18 Oct.18 Oct
T18The conversation layer is one small MCP server over the client's semantic layer; no UI of our own (was D7)Any chat surface can use it. Accepted with the other recommendations on 18 Oct.18 Oct
T19No checking of replies while they stream, for now (was D8)FNOL and clothing don't stream to the customer; revisit when an agent does. Accepted with the other recommendations on 18 Oct.18 Oct
T20Carry the work after results land, scoped: jobs once against ports, a five-method adapter per tool, a short supported list, steps 1–6 first (was D9)Keeps the loop's work bounded. Accepted with the other recommendations on 18 Oct.18 Oct
T21Trial two cost rules as a dbt package before choosing between dbt and our own runner (was D10)Decide on evidence. Accepted with the other recommendations on 18 Oct.18 Oct
T22The agent team may switch its own rules off, except rules the risk owner locks (was D11)Teams keep freedom; the org keeps its minimum. Accepted with the other recommendations on 18 Oct.18 Oct
T23Results published as OTel's gen_ai.evaluation.result, extended with scale, threshold, findings and rule identity (was D12)Any tool can read them without our code. Accepted with the other recommendations on 18 Oct.18 Oct
T24Session: keep five kinds of step; a guardrail block is a blocked decision; a person's reply is an assistant step named human; every step has an optional time; outcomes are resolved, handed_over, abandoned, errorCovers what ops asks without new kinds. From the tutorial review.18 Oct
T25Patterns: phrase matching first, AI judge later as an opt-in per rule; YAML keeps under/over, reports say missed/needless; consistency guards amounts and ids now, dates nextSame verdict every run until a team chooses to pay for a judge. From the tutorial review.18 Oct
T26Every rule takes a plain description; ready-made fact types replace hand-written regular expressionsAn ops person should be able to write and read the rules file. From the tutorial review.18 Oct
T27No hand-over proposal to OpenTelemetry until a second team needs it; meanwhile a team maps its own attribute in YAMLOne team's need is not yet a standard. From the tutorial review.18 Oct
T28Trace sources in this order: OTLP file (built), Phoenix, App Insights, LangfuseApp Insights is where the Azure clients and FNOL already are. From the tutorial review.18 Oct
T29Inline hooks: the loop only reports the moment (five calls to Hooks.run); each hook carries a matcher (tool name, pattern such as issue_*, or kind of tool such as irreversible); one decision type (allow, modify, block, escalate); first block wins. Placed by decorator or in YAML (at, tools, on_fail). Framework and gateway adapters only translate their hook into an Event.Borrowed from Claude Code's PreToolUse/PostToolUse matchers, the agent SDKs' guardrails and LiteLLM's pre/post-call guardrails. Targeting one tool never touches the loop; matching by kind covers new tools. Your question, 19 Oct.19 Oct
T30Checks that need the conversation's words run inside the agent's environment; only verdicts go to App Insights, content-free by default (counts and finding kinds, never quoted words or amounts). Content capture stays off in production; a dev deployment with synthetic customers may turn it on.Production traces with privacy on show what happened, not what was said (run 4b). Moving words into a trace store would fail a regulated client's review. Your choice, 19 Oct.19 Oct
T31A stack binding names stores by role (detail, numbers, history) and says what each can do; each ops question is routed to the cheapest role that can answer it, or reported as “can't be answered here” (was D15)Azure keeps everything in App Insights; the open stack splits Langfuse and Prometheus. Checks and questions must not change between them. Your choice, 16 Nov.16 Nov
T32Counters are added up by the Collector where there is one, by the store at question time when it is a single store that keeps every span, and never from sampled traces (was D16)A total counted from a sample is wrong, and nobody can see that it is.16 Nov
T33Every verdict is written as an event on the trace and as a counter, from one call (was D17)The event opens the conversation; the counter makes rates and alerts cheap and survives sampling.16 Nov
T34On Azure the default is App Insights alone; managed Prometheus is added when alerts must run more often than every 15 minutes, traces are sampled, or many agents share one board (was D18)Nothing extra to run until one of the three is true.16 Nov