Eval Harness Tutorial · From traces
The weekly report
The weekly report is the first thing an ops lead would read. It's built only from the verdicts in App Insights, never from conversations, so it holds no customer content. Every number is a count and every list is a filter: the same week always gives the same report.
What's on it
| Section | What it shows | Why it's honest |
|---|---|---|
| Rules | Per rule: pass, fail, nothing to check, could not judge; the fail rate; the change from the week before | The fail rate counts only conversations actually judged. Nothing to check and could-not-judge are never passes. |
| Saved questions | The six questions' answers, with the conversation, day and rule | The same definition as the App Insights queries and the chat (saved questions) |
| Requirements broken | Which spec requirements failed, and how often | A failure whose rule names no requirement is counted apart, as a gap in the rules file |
| Test data | A warning when test conversations are counted | Test verdicts are marked synthetic and left out unless asked for |
This week's report for FNOL
Built today from App Insights. Today FNOL's verdicts are all from test scenarios, so the warning is on. The one under-escalation is the deliberate demo verdict.
Three things the first build got wrong, fixed before you saw it
- It mixed two batches of verdicts written in different formats on the same afternoon. Every verdict now carries
evalharness.schema, a format version, so old and new can be told apart; and a period can start after a format change (--start). - With
--start, it compared against an unfair “week before” made of the old batch. A chosen period is now compared with nothing. - It said “no requirement was broken” while a rule had failed: that rule simply named no requirement. It now says so, and counts such failures apart.
Run it
.venv/bin/python scripts/weekly_report.py \
--resource-group <group> --workspace-name <workspace> \
--agent claims-fnol --out report.html # real customers only
# --include-synthetic to count test conversations tooThe counting, in real code
src/evalharness/report.py:113def weekly(
verdicts: Iterable[Verdict],
*,
start: float,
end: float,
previous: Iterable[Verdict] = (),
agent: str = "",
include_synthetic: bool = False,
) -> Report:
def keep(v: Verdict) -> bool:
return (include_synthetic or not v.synthetic) and (not agent or v.agent == agent)
week = [v for v in verdicts if start <= v.at < end and keep(v)]
before = [v for v in previous if keep(v)]
counts: dict[str, Counter[str]] = defaultdict(Counter)
for v in week:
counts[v.rule][v.label] += 1
earlier: dict[str, Counter[str]] = defaultdict(Counter)
for v in before:
earlier[v.rule][v.label] += 1
broken: Counter[str] = Counter()
for v in week:
if v.label == "fail":
broken.update(v.requirements)
return Report(
agent=agent,
start=start,
end=end,
rules=tuple(RuleLine(r, dict(counts[r]), dict(earlier.get(r, {}))) for r in sorted(counts)),
questions={q.name: tuple(v for v in week if matches(q, v)) for q in QUESTIONS},
requirements=dict(broken.most_common()),
unnamed=sum(1 for v in week if v.label == "fail" and not v.requirements),
verdicts=len(week),
unjudged=sum(1 for v in week if v.label == "skip"),
synthetic=include_synthetic,
)Your review · The weekly report
- Is this what an ops lead should see first on a Monday? What would you add, move or remove?
Reply in chat with the page name and the number, for example “session 2: …”.