Eval Harness Tutorial · From traces
How the rebuild works: three traps
Rebuilding sounds simple. On a real framework's traces, three things would have produced the wrong conversation:
| What we saw | What would go wrong | What the reader does |
|---|---|---|
| Every model call resends the whole conversation so far | Each customer message would appear many times | Takes only the customer's words after the model last spoke, once per trace |
| Pydantic AI sends tool results back as customer messages with no text | Extra empty “customer” turns | A customer message with no text is not a turn |
| Each tool call is recorded twice: in the model's output and as its own tool span | Every call counted twice | One step per call id; the result comes from the tool span when there is one |
Each trap is a test in tests/test_traces.py, alongside cases
built by hand: no conversation id, an agent that doesn't resend history, calls
recorded only in messages, and a failed tool.