cleandataengineering.aiEval Harness, drawn simply
You are here: From traces › Weekly report

Eval Harness Tutorial · From traces

The weekly report

The weekly report is the first thing an ops lead would read. It's built only from the verdicts in App Insights, never from conversations, so it holds no customer content. Every number is a count and every list is a filter: the same week always gives the same report.

App Insightsverdict events onlyKQLweekly(verdicts)counts and filters,no model, no contentA pageself-contained HTML,prints cleanlylaterTeams / emailTier 8
Read the verdict events, count them, write one page. Sending it to Teams or email comes later (Tier 8).

What's on it

SectionWhat it showsWhy it's honest
RulesPer rule: pass, fail, nothing to check, could not judge; the fail rate; the change from the week beforeThe fail rate counts only conversations actually judged. Nothing to check and could-not-judge are never passes.
Saved questionsThe six questions' answers, with the conversation, day and ruleThe same definition as the App Insights queries and the chat (saved questions)
Requirements brokenWhich spec requirements failed, and how oftenA failure whose rule names no requirement is counted apart, as a gap in the rules file
Test dataA warning when test conversations are countedTest verdicts are marked synthetic and left out unless asked for

This week's report for FNOL

Built today from App Insights. Today FNOL's verdicts are all from test scenarios, so the warning is on. The one under-escalation is the deliberate demo verdict.

Open the report on its own page
Three things the first build got wrong, fixed before you saw it
  1. It mixed two batches of verdicts written in different formats on the same afternoon. Every verdict now carries evalharness.schema, a format version, so old and new can be told apart; and a period can start after a format change (--start).
  2. With --start, it compared against an unfair “week before” made of the old batch. A chosen period is now compared with nothing.
  3. It said “no requirement was broken” while a rule had failed: that rule simply named no requirement. It now says so, and counts such failures apart.
Run it
.venv/bin/python scripts/weekly_report.py \
    --resource-group <group> --workspace-name <workspace> \
    --agent claims-fnol --out report.html        # real customers only
    # --include-synthetic to count test conversations too
The counting, in real codesrc/evalharness/report.py:113
def weekly(
    verdicts: Iterable[Verdict],
    *,
    start: float,
    end: float,
    previous: Iterable[Verdict] = (),
    agent: str = "",
    include_synthetic: bool = False,
) -> Report:
    def keep(v: Verdict) -> bool:
        return (include_synthetic or not v.synthetic) and (not agent or v.agent == agent)

    week = [v for v in verdicts if start <= v.at < end and keep(v)]
    before = [v for v in previous if keep(v)]
    counts: dict[str, Counter[str]] = defaultdict(Counter)
    for v in week:
        counts[v.rule][v.label] += 1
    earlier: dict[str, Counter[str]] = defaultdict(Counter)
    for v in before:
        earlier[v.rule][v.label] += 1
    broken: Counter[str] = Counter()
    for v in week:
        if v.label == "fail":
            broken.update(v.requirements)
    return Report(
        agent=agent,
        start=start,
        end=end,
        rules=tuple(RuleLine(r, dict(counts[r]), dict(earlier.get(r, {}))) for r in sorted(counts)),
        questions={q.name: tuple(v for v in week if matches(q, v)) for q in QUESTIONS},
        requirements=dict(broken.most_common()),
        unnamed=sum(1 for v in week if v.label == "fail" and not v.requirements),
        verdicts=len(week),
        unjudged=sum(1 for v in week if v.label == "skip"),
        synthetic=include_synthetic,
    )
Your review · The weekly report
  1. Is this what an ops lead should see first on a Monday? What would you add, move or remove?

Reply in chat with the page name and the number, for example “session 2: …”.