cleandataengineering.aiEval Harness, drawn simply
You are here: From traces › Verdicts into App Insights

Eval Harness Tutorial · From traces

Verdicts back into App Insights

Your decision (T30): checks that need the conversation's words run where the words already are, inside the agent's environment. Only the verdicts go to App Insights. Content capture stays off. This page shows it working on Azure.

INSIDE THE AGENT'S ENVIRONMENT (the words stay here)The conversationwords, amounts, toolarguments and resultsOur checksthe five patterns,with everything they needOtelEvents sinkverdict, counts, kinds,requirement idsverdicts onlyApp InsightsAppTraces, besidethe turn's traceSaved questionsas KQLDashboardsworkbook templateAlertsKQL alert rules
The left box never sends words anywhere. App Insights receives one small event per check per conversation, next to the turn's own trace.

What one verdict looks like

Each verdict is an OpenTelemetry gen_ai.evaluation.result event, the standard's own shape. Our extra fields go under evalharness.*. Here is the facts check failing on the made-up ₹15,000 from the consistency page. Note the explanation: counts and kinds only. The amount the agent made up is not sent.

AttributeSent
gen_ai.evaluation.namefacts_from_systems
gen_ai.evaluation.score.labelfail
gen_ai.evaluation.explanation1 of 2 held; unsupported_fact
evalharness.schema1
evalharness.verdictfail
evalharness.providerours
evalharness.version1
gen_ai.evaluation.score.value0.5
evalharness.threshold1.0
evalharness.raw_score0.5
evalharness.finding_kindsunsupported_fact
evalharness.findings1
evalharness.checkconsistency
evalharness.obligationsAAC-0029
evalharness.session.idconv-9

With explain="full" (only where the store may hold conversation content), the explanation would instead read: 1 of 2 held; amount “₹15,000” is in no earlier tool result.

The run on Azure

FNOL's 29 scenarios (48 conversations), checked inside FNOL's environment; only verdicts sent to App Insights, marked synthetic. The one fail is a deliberate demo verdict sent to prove the under_escalations query finds it. Every conversation got one verdict per rule, 241 in all. Counted back from App Insights with KQL:

Rulepassfailnot applicable
facts_from_systems17—31
handed_to_person48——
payout_approval481—
payout_needs_claim_lookup4—44
payout_promise_kept3—45
Not applicable is not a pass

A payout rule on a conversation with no payout had nothing to check. Counting that as a pass would make every report look better than it is, so it's labelled not_applicable. The labels this sink writes: pass, fail, not_applicable, skip, error.

Saved questions, as App Insights queries

Each saved question is generated into KQL from its one definition. This is under_escalations. On Azure it found exactly the one deliberate under-escalation, and the other questions found nothing.

to_kql(under_escalations): paste into App Insights › Logs
AppTraces
| where TimeGenerated > ago(7d)
| extend p = parse_json(Properties)
| where tostring(p['evalharness.check']) == 'decision_vs_should'
| where split(tostring(p['evalharness.finding_kinds']), ',') has 'under'
| where tostring(p['evalharness.synthetic']) != 'True'
| project TimeGenerated, OperationId,
    rule = tostring(p['gen_ai.evaluation.name']),
    session = tostring(p['evalharness.session.id']),
    requirements = tostring(p['evalharness.requirements'])
| order by TimeGenerated desc
def attributes(
    result: EvalResult,
    *,
    rule: Any = None,
    request: EvalRequest | None = None,
    explain: str = "counts",
    synthetic: bool = False,
    agent: str = "",
    share: float | None = None,
) -> dict[str, Any]:
    """The event's attributes: the conventions' names first, ours after."""
    a: dict[str, Any] = {
        "gen_ai.evaluation.name": result.evaluator,
        "gen_ai.evaluation.score.label": label(result),
        "gen_ai.evaluation.explanation": result.reason if explain == "full" else _counts(result),
        "evalharness.schema": SCHEMA,
        "evalharness.verdict": result.verdict,
        "evalharness.provider": result.provider,
        "evalharness.version": result.version,
    }
    if result.score is not None:
        a["gen_ai.evaluation.score.value"] = float(result.score)
    if result.threshold is not None:
        a["evalharness.threshold"] = float(result.threshold)
    if result.raw_score is not None:
        a["evalharness.raw_score"] = float(result.raw_score)
    if result.verdict == "error":
        a["error.type"] = "_OTHER"
    kinds = sorted({f.kind for f in result.findings})
    if kinds:
        a["evalharness.finding_kinds"] = ",".join(kinds)
        a["evalharness.findings"] = len(result.findings)
    if rule is not None:
        a["evalharness.check"] = getattr(rule, "kind", "")
        if getattr(rule, "requirements", ()):
            a["evalharness.requirements"] = ",".join(rule.requirements)
        if getattr(rule, "obligations", ()):
            a["evalharness.obligations"] = ",".join(rule.obligations)
    if request is not None and request.session is not None:
        a["evalharness.session.id"] = request.session.id
    if request is not None and request.meta.trace:
        a["evalharness.trace"] = request.meta.trace
    agent = agent or (request.meta.agent if request is not None else "")
    if agent:
        a["evalharness.agent"] = agent
    if synthetic:
        a["evalharness.synthetic"] = True
    if share is not None:
        a["evalharness.sample_share"] = float(share)
    return a
Decided · Verdicts into App Insights
  1. Is the content-free default right: counts and finding kinds, never the quoted words or amounts?
    Decided: Yes (19 Oct): the full explanation only where the store may hold content, such as a dev deployment.

What a team gets on top of this, tool by tool: on top of your tools.