Eval Harness Tutorial · From traces
From the agent's traces to a verdict
Most agents already write traces: a record of every piece of work, in a standard format called OpenTelemetry. If we can rebuild conversations from traces, a team adds no code to their agent to be checked. This page follows one real recording through every step.
A small refund agent built with Pydantic AI, a framework we didn't write, with its own built-in tracing switched on. Its model is scripted, so no AI is called and the recording is identical every time (scripts/record_genai_traces.py). It has three conversations. We follow the one that refunds without looking the order up.
1 · The agent runs, and writes spans
The whole recording is 16 spans. These are the ones for our conversation:
| Span | Operation | Status | Tool arguments |
|---|---|---|---|
chat scripted | chat | ok | — |
execute_tool issue_refund | execute_tool | ok | {"order_id":"42","amount":500} |
chat scripted | chat | ok | — |
invoke_agent refund-agent | invoke_agent | ok | — |
2 · Group spans into conversations
gen_ai.conversation.id; if missing, by session.id; if missing, each trace is its own conversation3 · Rebuild the steps
Inside a group, spans are put in time order and turned into steps. The customer's words come from the model's input messages; the replies from its output messages; tool calls and results from the tool spans. Three traps, all found on the real recording, are in how the rebuild works.
4 · Run the rule
The rule: issue_refund needs a successful get_order for the same order_id first.
verdict | fail |
score | 0.00 |
reason | 0 of 1 held; issue_refund without an earlier get_order for the same order_id |
evaluator | refund_needs_lookup |
provider | ours |
findings | missing_before: issue_refund without an earlier get_order for the same order_id |
5 · All three conversations
| Conversation | Size | refund_needs_lookup |
|---|---|---|
conv-lookup-first | 8 steps | pass |
conv-no-lookup | 4 steps | fail |
conv-tool-error | 4 steps | pass |
The one that looked the order up first passes. The one where the lookup failed passes too: it never refunded, so there was nothing to check.
- Is following one conversation from raw spans to verdict the right way to explain this, or too much detail?
Decided: Yes, keep the one-conversation walk-through. - Phoenix is next (it's where many teams' traces already are). Is there a store your clients use more, such as App Insights or Langfuse?
Decided: App Insights second (the Azure clients, and FNOL already writes there), then Langfuse.