Eval Harness Tutorial · Your review
Questions and decisions
9 of 41 questions are open. Each links back to the page that explains it. Open questions are answered in chat with the page name and the number, for example “session 2: yes, add times”. To change a decision, say so the same way.
The whole picture
- Is the five-step picture clear enough to explain to an ops person in two minutes? If not, which step is unclear?
Decided: Yes. The five steps stay the explanation. - Is anything missing that an ops team would expect, such as a weekly report or alerts?
Decided: Yes: a weekly report and alerts. Both belong to step 5 (decide and record), now drawn there as not built yet.
The big picture
- Does this drawing show what we're trying to do? What would you add or remove before showing it to a client?
Decided: Yes. For a client, drop the tier labels and keep the four ways in.
Offline, inline, online
- Is the three-timings explanation clear enough for an ops lead deciding where a rule should run?
Decided: Yes. The comparison table is the one-slide version.
The whole chain
- Is this the drawing you would show a CTO, or should the client version hide the status colours and show only the five rows?
- Row 3 lists eight stores. Which three matter most for the clients you expect first? My suggestion: App Insights, Databricks MLflow traces, then CloudWatch.
Reply in chat with the page name and the number, for example “session 2: …”.
One store or two
- Should a binding name stores by role (detail, numbers, history), as drawn?
Decided: Yes (T31). - Who adds up the counters?
Decided: The Collector where there is one; the store at question time when it is a single store that keeps every span; never from a sample (T32). - Should every verdict be written as an event and a counter from one call?
Decided: Yes (T33). - Which shape do we present first to an Azure client?
Decided: Shape A, App Insights alone, with shape B as the stated next step and its three triggers (T34).
One question, three languages
- The counters are named after the GenAI conventions' duration histogram. Is that the name you want clients to see, or should we use the name the Collector's own span-metrics step produces? My suggestion: keep the conventions' name and let the binding override it.
- What comes next: the Grafana board for these nine metrics, or metrics for cost (group C)? My suggestion: the board first, so there is something to show.
Reply in chat with the page name and the number, for example “session 2: …”.
The Session shape
- Are the five kinds of step enough? Is there anything an ops person asks about that isn't one of them (for example a guardrail blocking a reply, or a person replying after a hand-over)?
Decided: Keep five kinds. A guardrail blocking a reply is adecisionnamedblocked. A person's reply after a hand-over is anassistantstep namedhuman, which checks on the agent's own words skip. Built. - Should each step carry its time? It costs little, and enables “how long did the customer wait” rules.
Decided: Yes. Every step has an optional time (at); sessions rebuilt from traces fill it in. Built. - For
outcome: which words should a conversation end with in your world (resolved, abandoned, escalated, something else)?
Decided:resolved,handed_over,abandoned,error. A host may use its own words; reports group unknown ones as “other”.
Hooks in your own loop
- Is “the loop reports the moment, the hook says what it's for” simple enough to explain to an agent team?
Decided: Recommended: yes, and lead with matching by kind of tool (irreversible, write), because it covers new tools without new rules.
Who may change a rule
- Is 0.9 precision on 50 conversations the right bar to block, or should it differ per kind of rule (a PII mask vs. stopping a refund)?
Reply in chat with the page name and the number, for example “session 2: …”.
Cost rules
- Which cost rules matter most to your clients? This sets the order of the remaining 30.
Reply in chat with the page name and the number, for example “session 2: …”.
On top of your tools
- Which saved questions would an ops lead ask first? Six are built in; what's missing from the list?
Decided: Under- and over-escalations and promises not kept first, then actions without a prerequisite. Add next: “a tool kept failing and nobody took over”. - For a LangSmith client, is “results as feedback, alerts and saved filters from our templates, chat through MCP” the right offer?
Decided: Yes: results as feedback on the same run, alert and saved-filter templates, the chat through MCP. No LangSmith dashboard template.
requires_before
- In your ops world, what are the top three “A before B” rules? Do they all fit this shape?
Decided: Starting set: pay only after reading the claim; cancel or refund only after reading the order; change account details only after verifying the customer. All three fit this shape.
argument_limit
- Are limits in your world usually on the call itself, or on a record the agent read first (like FNOL)?
Decided: Both happen. A limit on the call usesargument_limit; a limit on a record the agent read usesdecision_vs_shouldwithresult_of, as FNOL does. No change needed.
claim_needs_action
- Is phrase matching good enough to start with, or would you rather pay for an AI judge from day one?
Decided: Phrases first. The AI-judge version comes later as an opt-in per rule, for teams that will pay for it. - Which “done” claims matter most in your world (paid, refunded, cancelled, booked, escalated)?
Decided: Paid, refunded, cancelled and handed over first; booked when a booking agent comes along.
decision_vs_should
- Is “under” and “over” the right way to talk about this to ops people, or would they say it differently?
Decided: Keepunderandoverin the YAML; reports say “missed” and “needless”.
consistency
- Which facts would an ops team most want guarded: amounts, ids, dates, policy terms?
Decided: Amounts and ids now. Dates next, because they need one format before they can be compared.
The FNOL rules file
- Could an ops person write this file without help? Which line would they get stuck on?
Decided: Not yet: the regular expressions underconsistencyare where they'd stop. Next: ready-made fact types (an Indian-rupee amount, a claim id) so nobody writes a regular expression. - Is anything missing that an FNOL ops lead would want checked?
Decided: Nothing added until a real FNOL ops lead asks; rules come from the spec. - The comments explain each rule. Would you rather have a separate plain-English description field on each rule, which reports would show?
Decided: Yes. Every rule takes an optionaldescription, and the FNOL example now has one per rule. Built.
From traces
- Is following one conversation from raw spans to verdict the right way to explain this, or too much detail?
Decided: Yes, keep the one-conversation walk-through. - Phoenix is next (it's where many teams' traces already are). Is there a store your clients use more, such as App Insights or Langfuse?
Decided: App Insights second (the Azure clients, and FNOL already writes there), then Langfuse.
Gaps in the standard
- Should we propose a hand-over field to OpenTelemetry now, or wait until a second team needs it?
Decided: Wait for a second team. Meanwhile a team maps its own attribute to a decision in YAML.
Real traces
- For content checks on a production agent, where should they run?
Decided: Inside the agent's environment, where the words already are; only verdicts go to App Insights, so no customer words leave. Content capture stays off in production (T30, 19 Oct). Built: see the next page.
Verdicts into App Insights
- Is the content-free default right: counts and finding kinds, never the quoted words or amounts?
Decided: Yes (19 Oct): the full explanation only where the store may hold content, such as a dev deployment.
The weekly report
- Is this what an ops lead should see first on a Monday? What would you add, move or remove?
Reply in chat with the page name and the number, for example “session 2: …”.
The catalog
- Is “any agent / by kind of agent / for one business” the right way to sort checks for a client conversation?
Decided: Yes, keep the three layers. - Which gaps should we close first: the core obligations, the cost rules, or more domain requirements?
Decided: Domain requirements first (that's what a client pays for), then the cost rules (Tier 5), then the core obligations that can be checked after the fact.
Ops questions
- Which 10 should the first dashboard pack answer? My suggestion: A1, A2, B1, C1, C2, D1, E2, F1, G1 and K1, because they need span metadata only and work on every tool.
- Is anything an ops lead asks every week missing from these 12 groups?
Reply in chat with the page name and the number, for example “session 2: …”.
Requirements
- Should every rule be required to name at least one requirement, so no rule exists without a reason?
Decided: Yes, as a warning rather than a refusal until a team has a spec. FNOL and the clothing agent already comply. - Which untested FNOL requirements matter most to check after the fact?
Decided: P-PAYOUT-OWED (pay when asked; a status question issues nothing), P-OWN-WORDS (act only on a named claim), P-OWNERSHIP, and the “never a date” half of P-CLAIM-STATUS.