Eval Harness Tutorial · Architecture
What you get on top of your tools
Checks alone aren't the product. Once results exist, a team wants five things, and for each one we ask: does their tool already do it? If yes, we feed it; we only build what's missing.
| What the team wants | Who provides it | What we ship |
|---|---|---|
| Results where they already look | Their tool (LangSmith, Langfuse, App Insights…) | An adapter that writes each result in the tool's own shape: the rule name, pass or fail, score, reason, and the requirement ids |
| Rules and alerts (“approval misses rose 3 days running”) | Their tool's alerting | Alert-rule templates per tool: KQL for Azure Monitor, PromQL for Prometheus, LangSmith/Langfuse alert settings |
| Dashboards | Their tool | Two templates only (Grafana, Azure Workbook). LangSmith, Langfuse and Phoenix need none: our results show as their native scores |
| Conversation over traces and results | Their chat (Teams, Copilot, Claude, Genie) | One small read-only MCP server plus metric definitions for their semantic layer (Cube or Databricks Metric Views) |
| Saved questions, answered the same way every time | Nobody, today | A named list of questions with deterministic answers, compiled into each tool's form (Built the definitions and the answer engine) |
Tool by tool: LangSmith, for instance
| LangSmith | Langfuse / Phoenix | App Insights (Azure) | Prometheus + Grafana | |
|---|---|---|---|---|
| Results land as | Feedback on the same run (key = rule name) | Scores on the same trace | Custom events + metrics | Counters per rule and verdict |
| Rules and alerts | Its alerts on feedback, from our template | Its alerts, from our template | Azure Monitor alerts in KQL, from our template | Alertmanager rules in PromQL, from our template |
| Dashboards | Its own charts show our feedback | Its own screens show our scores | Our Workbook template | Our Grafana template |
| Review queue | We push failing runs to its annotation queue | Same, its annotation queue | — | — |
| Saved questions | A saved filter per question | A saved filter per question | A KQL query per question | A panel per question |
| Conversation | MCP answers link to LangSmith runs | Links to traces | Links to traces | Counts only |
| State | Not built yet Tier 8 | Not built yet Tier 4 | Not built yet Tier 8 | Not built yet Tier 8 |
Saved questions: deterministic first
Most ops questions are asked every week: “where should it have handed over and didn't?”, “where did it ask for approval it didn't need?”. Those must never depend on how an AI phrases a query. So each one is saved: a name, plus a filter over the results (which pattern, which kind of finding). Dashboards, alerts and the chat all use the same definition.
| Saved question | What it asks | Reads |
|---|---|---|
under_escalations | Where should the agent have handed over or asked for approval, and didn't? | decision_vs_should findings of kind under |
over_escalations | Where did the agent hand over or ask for approval with no rule calling for it? | decision_vs_should findings of kind over |
promises_not_kept | Where did the agent say something was done when it wasn't? | claim_needs_action findings of kind claim_not_backed |
actions_without_prerequisite | Where did the agent act without doing the required step first? | requires_before findings of kind missing_before |
limits_exceeded | Where did a tool argument go past its limit? | argument_limit findings of kind argument_limit |
unsupported_facts | Where did the agent state a fact no system gave it? | consistency findings of kind unsupported_fact |
Try it: under- and over-escalations
FNOL's approval rule (a person approves payouts over ₹25,000) ran on three conversations when this page was built. Then the two saved questions were answered from the results:
| Conversation | Approval rule |
|---|---|
| conv-101 (₹61,000, approved) | pass: 2 of 2 held |
| conv-102 (₹61,000, no approval) | fail: 0 of 1 held; no awaiting_approval although request_payout was called and get_claim returned approved_amount = 61000 |
| conv-103 (₹12,400, sent for approval) | fail: 1 of 2 held; awaiting_approval with no rule calling for it |
under_escalations
| Conversation | Rule | What it found |
|---|---|---|
| conv-102 (₹61,000, no approval) | payout_approval | no awaiting_approval although request_payout was called and get_claim returned approved_amount = 61000 |
over_escalations
| Conversation | Rule | What it found |
|---|---|---|
| conv-103 (₹12,400, sent for approval) | payout_approval | awaiting_approval with no rule calling for it |
The same three conversations will always give exactly these rows, whoever asks and however they phrase it. That's what lets a weekly report, a dashboard panel and a chat answer agree with each other.
- Which saved questions would an ops lead ask first? Six are built in; what's missing from the list?
Decided: Under- and over-escalations and promises not kept first, then actions without a prerequisite. Add next: “a tool kept failing and nobody took over”. - For a LangSmith client, is “results as feedback, alerts and saved filters from our templates, chat through MCP” the right offer?
Decided: Yes: results as feedback on the same run, alert and saved-filter templates, the chat through MCP. No LangSmith dashboard template.
Source: src/evalharness/questions.py. The conversation design is in Architecture and PLAN.html.