cleandataengineering.aiEval Harness, drawn simply
You are here: From traces › Three traps

Eval Harness Tutorial · From traces

How the rebuild works: three traps

Rebuilding sounds simple. On a real framework's traces, three things would have produced the wrong conversation:

What we sawWhat would go wrongWhat the reader does
Every model call resends the whole conversation so farEach customer message would appear many timesTakes only the customer's words after the model last spoke, once per trace
Pydantic AI sends tool results back as customer messages with no textExtra empty “customer” turnsA customer message with no text is not a turn
Each tool call is recorded twice: in the model's output and as its own tool spanEvery call counted twiceOne step per call id; the result comes from the tool span when there is one

Each trap is a test in tests/test_traces.py, alongside cases built by hand: no conversation id, an agent that doesn't resend history, calls recorded only in messages, and a failed tool.