cleandataengineering.aiEval Harness, drawn simply
You are here: From traces

Eval Harness Tutorial · From traces

From the agent's traces to a verdict

Most agents already write traces: a record of every piece of work, in a standard format called OpenTelemetry. If we can rebuild conversations from traces, a team adds no code to their agent to be checked. This page follows one real recording through every step.

The recording

A small refund agent built with Pydantic AI, a framework we didn't write, with its own built-in tracing switched on. Its model is scripted, so no AI is called and the recording is identical every time (scripts/record_genai_traces.py). It has three conversations. We follow the one that refunds without looking the order up.

1 · The agent runs, and writes spans

invoke_agent refund-agentone agent runchatmodel asks for issue_refundexecute_tool issue_refundarguments + resultchatmodel writes the replyA trace is a tree of spans: one span per piece of work,each with a name, a start and end time, and attributes.gen_ai.input.messages / gen_ai.output.messagesgen_ai.tool.name, gen_ai.tool.call.arguments, .resultevery span: gen_ai.conversation.id = conv-no-lookup
One agent run. The names in code font are the standard's attribute names: any framework that follows the standard writes the same ones.

The whole recording is 16 spans. These are the ones for our conversation:

SpanOperationStatusTool arguments
chat scriptedchatok—
execute_tool issue_refundexecute_toolok{"order_id":"42","amount":500}
chat scriptedchatok—
invoke_agent refund-agentinvoke_agentok—

2 · Group spans into conversations

Goes inAll spans, from every conversation
What happensGroup by gen_ai.conversation.id; if missing, by session.id; if missing, each trace is its own conversation
Comes out3 groups, one per conversation

3 · Rebuild the steps

Inside a group, spans are put in time order and turned into steps. The customer's words come from the model's input messages; the replies from its output messages; tool calls and results from the tool spans. Three traps, all found on the real recording, are in how the rebuild works.

CUSTOMERAGENTTOOLS (its systems)Customer“Refund order 42 now.”calls issue_refundorder_id=42, amount=500issue_refund returnsorder_id: 42, refunded: 500Agent says“Done, your refund of 500 is onits way.”
The rebuilt session: the agent refunds with no lookup first.

4 · Run the rule

The rule: issue_refund needs a successful get_order for the same order_id first.

CUSTOMERAGENTTOOLS (its systems)Customer“Refund order 42 now.”✗ calls issue_refundorder_id=42, amount=500issue_refund returnsorder_id: 42, refunded: 500Agent says“Done, your refund of 500 is onits way.”✗ step 2: issue_refund without an earlier get_order for the same order_id
The check flags the refund.
verdictfail
score0.00
reason0 of 1 held; issue_refund without an earlier get_order for the same order_id
evaluatorrefund_needs_lookup
providerours
findingsmissing_before: issue_refund without an earlier get_order for the same order_id

5 · All three conversations

ConversationSizerefund_needs_lookup
conv-lookup-first8 stepspass
conv-no-lookup4 stepsfail
conv-tool-error4 stepspass

The one that looked the order up first passes. The one where the lookup failed passes too: it never refunded, so there was nothing to check.

Decided · From traces
  1. Is following one conversation from raw spans to verdict the right way to explain this, or too much detail?
    Decided: Yes, keep the one-conversation walk-through.
  2. Phoenix is next (it's where many teams' traces already are). Is there a store your clients use more, such as App Insights or Langfuse?
    Decided: App Insights second (the Azure clients, and FNOL already writes there), then Langfuse.

Look inside