cleandataengineering.aiEval Harness, drawn simply
You are here: Architecture › On top of your tools

Eval Harness Tutorial · Architecture

What you get on top of your tools

Checks alone aren't the product. Once results exist, a team wants five things, and for each one we ask: does their tool already do it? If yes, we feed it; we only build what's missing.

What the team wantsWho provides itWhat we ship
Results where they already lookTheir tool (LangSmith, Langfuse, App Insights…)An adapter that writes each result in the tool's own shape: the rule name, pass or fail, score, reason, and the requirement ids
Rules and alerts (“approval misses rose 3 days running”)Their tool's alertingAlert-rule templates per tool: KQL for Azure Monitor, PromQL for Prometheus, LangSmith/Langfuse alert settings
DashboardsTheir toolTwo templates only (Grafana, Azure Workbook). LangSmith, Langfuse and Phoenix need none: our results show as their native scores
Conversation over traces and resultsTheir chat (Teams, Copilot, Claude, Genie)One small read-only MCP server plus metric definitions for their semantic layer (Cube or Databricks Metric Views)
Saved questions, answered the same way every timeNobody, todayA named list of questions with deterministic answers, compiled into each tool's form (Built the definitions and the answer engine)

Tool by tool: LangSmith, for instance

LangSmithLangfuse / PhoenixApp Insights (Azure)Prometheus + Grafana
Results land asFeedback on the same run (key = rule name)Scores on the same traceCustom events + metricsCounters per rule and verdict
Rules and alertsIts alerts on feedback, from our templateIts alerts, from our templateAzure Monitor alerts in KQL, from our templateAlertmanager rules in PromQL, from our template
DashboardsIts own charts show our feedbackIts own screens show our scoresOur Workbook templateOur Grafana template
Review queueWe push failing runs to its annotation queueSame, its annotation queue——
Saved questionsA saved filter per questionA saved filter per questionA KQL query per questionA panel per question
ConversationMCP answers link to LangSmith runsLinks to tracesLinks to tracesCounts only
StateNot built yet Tier 8Not built yet Tier 4Not built yet Tier 8Not built yet Tier 8

Saved questions: deterministic first

Most ops questions are asked every week: “where should it have handed over and didn't?”, “where did it ask for approval it didn't need?”. Those must never depend on how an AI phrases a query. So each one is saved: a name, plus a filter over the results (which pattern, which kind of finding). Dashboards, alerts and the chat all use the same definition.

Someone asks“Show last week'sunder-escalations”1 · A saved question?matched by name:same answer every timeno2 · A defined metric?picked from the semanticlayer; never free SQLno3 · Say so“I don't have that.Closest: …”yesyesAnswer with evidencerows, the rule and requirement they came from, links to the traces
The chat tries the saved questions first, then the defined metrics, and otherwise says it doesn't know. It never writes its own SQL or KQL.
Saved questionWhat it asksReads
under_escalationsWhere should the agent have handed over or asked for approval, and didn't?decision_vs_should findings of kind under
over_escalationsWhere did the agent hand over or ask for approval with no rule calling for it?decision_vs_should findings of kind over
promises_not_keptWhere did the agent say something was done when it wasn't?claim_needs_action findings of kind claim_not_backed
actions_without_prerequisiteWhere did the agent act without doing the required step first?requires_before findings of kind missing_before
limits_exceededWhere did a tool argument go past its limit?argument_limit findings of kind argument_limit
unsupported_factsWhere did the agent state a fact no system gave it?consistency findings of kind unsupported_fact

Try it: under- and over-escalations

FNOL's approval rule (a person approves payouts over ₹25,000) ran on three conversations when this page was built. Then the two saved questions were answered from the results:

ConversationApproval rule
conv-101 (₹61,000, approved)pass: 2 of 2 held
conv-102 (₹61,000, no approval)fail: 0 of 1 held; no awaiting_approval although request_payout was called and get_claim returned approved_amount = 61000
conv-103 (₹12,400, sent for approval)fail: 1 of 2 held; awaiting_approval with no rule calling for it

under_escalations

ConversationRuleWhat it found
conv-102 (₹61,000, no approval)payout_approvalno awaiting_approval although request_payout was called and get_claim returned approved_amount = 61000

over_escalations

ConversationRuleWhat it found
conv-103 (₹12,400, sent for approval)payout_approvalawaiting_approval with no rule calling for it
Why this matters

The same three conversations will always give exactly these rows, whoever asks and however they phrase it. That's what lets a weekly report, a dashboard panel and a chat answer agree with each other.

Decided · On top of your tools
  1. Which saved questions would an ops lead ask first? Six are built in; what's missing from the list?
    Decided: Under- and over-escalations and promises not kept first, then actions without a prerequisite. Add next: “a tool kept failing and nobody took over”.
  2. For a LangSmith client, is “results as feedback, alerts and saved filters from our templates, chat through MCP” the right offer?
    Decided: Yes: results as feedback on the same run, alert and saved-filter templates, the chat through MCP. No LangSmith dashboard template.

Source: src/evalharness/questions.py. The conversation design is in Architecture and PLAN.html.