cleandataengineering.aiEval Harness, drawn simply
You are here: Your review

Eval Harness Tutorial · Your review

Questions and decisions

9 of 41 questions are open. Each links back to the page that explains it. Open questions are answered in chat with the page name and the number, for example “session 2: yes, add times”. To change a decision, say so the same way.

The whole picture
  1. Is the five-step picture clear enough to explain to an ops person in two minutes? If not, which step is unclear?
    Decided: Yes. The five steps stay the explanation.
  2. Is anything missing that an ops team would expect, such as a weekly report or alerts?
    Decided: Yes: a weekly report and alerts. Both belong to step 5 (decide and record), now drawn there as not built yet.
The big picture
  1. Does this drawing show what we're trying to do? What would you add or remove before showing it to a client?
    Decided: Yes. For a client, drop the tier labels and keep the four ways in.
Offline, inline, online
  1. Is the three-timings explanation clear enough for an ops lead deciding where a rule should run?
    Decided: Yes. The comparison table is the one-slide version.
The whole chain
  1. Is this the drawing you would show a CTO, or should the client version hide the status colours and show only the five rows?
  2. Row 3 lists eight stores. Which three matter most for the clients you expect first? My suggestion: App Insights, Databricks MLflow traces, then CloudWatch.

Reply in chat with the page name and the number, for example “session 2: …”.

One store or two
  1. Should a binding name stores by role (detail, numbers, history), as drawn?
    Decided: Yes (T31).
  2. Who adds up the counters?
    Decided: The Collector where there is one; the store at question time when it is a single store that keeps every span; never from a sample (T32).
  3. Should every verdict be written as an event and a counter from one call?
    Decided: Yes (T33).
  4. Which shape do we present first to an Azure client?
    Decided: Shape A, App Insights alone, with shape B as the stated next step and its three triggers (T34).
One question, three languages
  1. The counters are named after the GenAI conventions' duration histogram. Is that the name you want clients to see, or should we use the name the Collector's own span-metrics step produces? My suggestion: keep the conventions' name and let the binding override it.
  2. What comes next: the Grafana board for these nine metrics, or metrics for cost (group C)? My suggestion: the board first, so there is something to show.

Reply in chat with the page name and the number, for example “session 2: …”.

The Session shape
  1. Are the five kinds of step enough? Is there anything an ops person asks about that isn't one of them (for example a guardrail blocking a reply, or a person replying after a hand-over)?
    Decided: Keep five kinds. A guardrail blocking a reply is a decision named blocked. A person's reply after a hand-over is an assistant step named human, which checks on the agent's own words skip. Built.
  2. Should each step carry its time? It costs little, and enables “how long did the customer wait” rules.
    Decided: Yes. Every step has an optional time (at); sessions rebuilt from traces fill it in. Built.
  3. For outcome: which words should a conversation end with in your world (resolved, abandoned, escalated, something else)?
    Decided: resolved, handed_over, abandoned, error. A host may use its own words; reports group unknown ones as “other”.
Hooks in your own loop
  1. Is “the loop reports the moment, the hook says what it's for” simple enough to explain to an agent team?
    Decided: Recommended: yes, and lead with matching by kind of tool (irreversible, write), because it covers new tools without new rules.
Who may change a rule
  1. Is 0.9 precision on 50 conversations the right bar to block, or should it differ per kind of rule (a PII mask vs. stopping a refund)?

Reply in chat with the page name and the number, for example “session 2: …”.

Cost rules
  1. Which cost rules matter most to your clients? This sets the order of the remaining 30.

Reply in chat with the page name and the number, for example “session 2: …”.

On top of your tools
  1. Which saved questions would an ops lead ask first? Six are built in; what's missing from the list?
    Decided: Under- and over-escalations and promises not kept first, then actions without a prerequisite. Add next: “a tool kept failing and nobody took over”.
  2. For a LangSmith client, is “results as feedback, alerts and saved filters from our templates, chat through MCP” the right offer?
    Decided: Yes: results as feedback on the same run, alert and saved-filter templates, the chat through MCP. No LangSmith dashboard template.
requires_before
  1. In your ops world, what are the top three “A before B” rules? Do they all fit this shape?
    Decided: Starting set: pay only after reading the claim; cancel or refund only after reading the order; change account details only after verifying the customer. All three fit this shape.
argument_limit
  1. Are limits in your world usually on the call itself, or on a record the agent read first (like FNOL)?
    Decided: Both happen. A limit on the call uses argument_limit; a limit on a record the agent read uses decision_vs_should with result_of, as FNOL does. No change needed.
claim_needs_action
  1. Is phrase matching good enough to start with, or would you rather pay for an AI judge from day one?
    Decided: Phrases first. The AI-judge version comes later as an opt-in per rule, for teams that will pay for it.
  2. Which “done” claims matter most in your world (paid, refunded, cancelled, booked, escalated)?
    Decided: Paid, refunded, cancelled and handed over first; booked when a booking agent comes along.
decision_vs_should
  1. Is “under” and “over” the right way to talk about this to ops people, or would they say it differently?
    Decided: Keep under and over in the YAML; reports say “missed” and “needless”.
consistency
  1. Which facts would an ops team most want guarded: amounts, ids, dates, policy terms?
    Decided: Amounts and ids now. Dates next, because they need one format before they can be compared.
The FNOL rules file
  1. Could an ops person write this file without help? Which line would they get stuck on?
    Decided: Not yet: the regular expressions under consistency are where they'd stop. Next: ready-made fact types (an Indian-rupee amount, a claim id) so nobody writes a regular expression.
  2. Is anything missing that an FNOL ops lead would want checked?
    Decided: Nothing added until a real FNOL ops lead asks; rules come from the spec.
  3. The comments explain each rule. Would you rather have a separate plain-English description field on each rule, which reports would show?
    Decided: Yes. Every rule takes an optional description, and the FNOL example now has one per rule. Built.
From traces
  1. Is following one conversation from raw spans to verdict the right way to explain this, or too much detail?
    Decided: Yes, keep the one-conversation walk-through.
  2. Phoenix is next (it's where many teams' traces already are). Is there a store your clients use more, such as App Insights or Langfuse?
    Decided: App Insights second (the Azure clients, and FNOL already writes there), then Langfuse.
Gaps in the standard
  1. Should we propose a hand-over field to OpenTelemetry now, or wait until a second team needs it?
    Decided: Wait for a second team. Meanwhile a team maps its own attribute to a decision in YAML.
Real traces
  1. For content checks on a production agent, where should they run?
    Decided: Inside the agent's environment, where the words already are; only verdicts go to App Insights, so no customer words leave. Content capture stays off in production (T30, 19 Oct). Built: see the next page.
Verdicts into App Insights
  1. Is the content-free default right: counts and finding kinds, never the quoted words or amounts?
    Decided: Yes (19 Oct): the full explanation only where the store may hold content, such as a dev deployment.
The weekly report
  1. Is this what an ops lead should see first on a Monday? What would you add, move or remove?

Reply in chat with the page name and the number, for example “session 2: …”.

The catalog
  1. Is “any agent / by kind of agent / for one business” the right way to sort checks for a client conversation?
    Decided: Yes, keep the three layers.
  2. Which gaps should we close first: the core obligations, the cost rules, or more domain requirements?
    Decided: Domain requirements first (that's what a client pays for), then the cost rules (Tier 5), then the core obligations that can be checked after the fact.
Ops questions
  1. Which 10 should the first dashboard pack answer? My suggestion: A1, A2, B1, C1, C2, D1, E2, F1, G1 and K1, because they need span metadata only and work on every tool.
  2. Is anything an ops lead asks every week missing from these 12 groups?

Reply in chat with the page name and the number, for example “session 2: …”.

Requirements
  1. Should every rule be required to name at least one requirement, so no rule exists without a reason?
    Decided: Yes, as a warning rather than a refusal until a team has a spec. FNOL and the clothing agent already comply.
  2. Which untested FNOL requirements matter most to check after the fact?
    Decided: P-PAYOUT-OWED (pay when asked; a status question issues nothing), P-OWN-WORDS (act only on a named claim), P-OWNERSHIP, and the “never a date” half of P-CLAIM-STATUS.