Eval Harness Tutorial · From traces
Verdicts back into App Insights
Your decision (T30): checks that need the conversation's words run where the words already are, inside the agent's environment. Only the verdicts go to App Insights. Content capture stays off. This page shows it working on Azure.
What one verdict looks like
Each verdict is an OpenTelemetry gen_ai.evaluation.result event,
the standard's own shape. Our extra fields go under evalharness.*. Here is
the facts check failing on the made-up ₹15,000 from the consistency page.
Note the explanation: counts and kinds only. The amount the agent made up is not sent.
| Attribute | Sent |
|---|---|
gen_ai.evaluation.name | facts_from_systems |
gen_ai.evaluation.score.label | fail |
gen_ai.evaluation.explanation | 1 of 2 held; unsupported_fact |
evalharness.schema | 1 |
evalharness.verdict | fail |
evalharness.provider | ours |
evalharness.version | 1 |
gen_ai.evaluation.score.value | 0.5 |
evalharness.threshold | 1.0 |
evalharness.raw_score | 0.5 |
evalharness.finding_kinds | unsupported_fact |
evalharness.findings | 1 |
evalharness.check | consistency |
evalharness.obligations | AAC-0029 |
evalharness.session.id | conv-9 |
With explain="full" (only where the store may hold conversation
content), the explanation would instead read: 1 of 2 held; amount “₹15,000” is in no earlier tool result.
The run on Azure
FNOL's 29 scenarios (48 conversations), checked inside FNOL's environment; only verdicts sent to App Insights, marked synthetic. The one fail is a deliberate demo verdict sent to prove the under_escalations query finds it. Every conversation got one verdict per rule, 241 in all. Counted back from App Insights with KQL:
| Rule | pass | fail | not applicable |
|---|---|---|---|
facts_from_systems | 17 | — | 31 |
handed_to_person | 48 | — | — |
payout_approval | 48 | 1 | — |
payout_needs_claim_lookup | 4 | — | 44 |
payout_promise_kept | 3 | — | 45 |
A payout rule on a conversation with no payout had nothing to check. Counting that as a pass would make every report look better than it is, so it's labelled not_applicable. The labels this sink writes: pass, fail, not_applicable, skip, error.
Saved questions, as App Insights queries
Each saved question is generated into KQL from its one definition. This is
under_escalations. On Azure it found exactly the one deliberate under-escalation, and the
other questions found nothing.
AppTraces
| where TimeGenerated > ago(7d)
| extend p = parse_json(Properties)
| where tostring(p['evalharness.check']) == 'decision_vs_should'
| where split(tostring(p['evalharness.finding_kinds']), ',') has 'under'
| where tostring(p['evalharness.synthetic']) != 'True'
| project TimeGenerated, OperationId,
rule = tostring(p['gen_ai.evaluation.name']),
session = tostring(p['evalharness.session.id']),
requirements = tostring(p['evalharness.requirements'])
| order by TimeGenerated descsrc/evalharness/sinks/otel_events.py:83def attributes(
result: EvalResult,
*,
rule: Any = None,
request: EvalRequest | None = None,
explain: str = "counts",
synthetic: bool = False,
agent: str = "",
share: float | None = None,
) -> dict[str, Any]:
"""The event's attributes: the conventions' names first, ours after."""
a: dict[str, Any] = {
"gen_ai.evaluation.name": result.evaluator,
"gen_ai.evaluation.score.label": label(result),
"gen_ai.evaluation.explanation": result.reason if explain == "full" else _counts(result),
"evalharness.schema": SCHEMA,
"evalharness.verdict": result.verdict,
"evalharness.provider": result.provider,
"evalharness.version": result.version,
}
if result.score is not None:
a["gen_ai.evaluation.score.value"] = float(result.score)
if result.threshold is not None:
a["evalharness.threshold"] = float(result.threshold)
if result.raw_score is not None:
a["evalharness.raw_score"] = float(result.raw_score)
if result.verdict == "error":
a["error.type"] = "_OTHER"
kinds = sorted({f.kind for f in result.findings})
if kinds:
a["evalharness.finding_kinds"] = ",".join(kinds)
a["evalharness.findings"] = len(result.findings)
if rule is not None:
a["evalharness.check"] = getattr(rule, "kind", "")
if getattr(rule, "requirements", ()):
a["evalharness.requirements"] = ",".join(rule.requirements)
if getattr(rule, "obligations", ()):
a["evalharness.obligations"] = ",".join(rule.obligations)
if request is not None and request.session is not None:
a["evalharness.session.id"] = request.session.id
if request is not None and request.meta.trace:
a["evalharness.trace"] = request.meta.trace
agent = agent or (request.meta.agent if request is not None else "")
if agent:
a["evalharness.agent"] = agent
if synthetic:
a["evalharness.synthetic"] = True
if share is not None:
a["evalharness.sample_share"] = float(share)
return a- Is the content-free default right: counts and finding kinds, never the quoted words or amounts?
Decided: Yes (19 Oct): the full explanation only where the store may hold content, such as a dev deployment.
What a team gets on top of this, tool by tool: on top of your tools.