Eval Harness Tutorial · Sessions
What a session is
A session is one whole conversation between a customer and the agent, written down as a list of steps in the order they happened. It's the one thing every conversation-level check reads.
Here is a real one, in FNOL's words. A customer asks for a payout; the agent reads the claim, pays it, and says so:
The five kinds of step
| Kind | What it records | Drawn as |
|---|---|---|
user | What the customer typed | left lane |
assistant | What the agent replied. A person's reply after a hand-over is one too, named human, and checks on the agent's own words skip it. | middle lane |
tool_call | The agent asked a system to do something: its name and arguments | right lane, dark |
tool_result | What the system answered, or that it failed | right lane, light |
decision | Something decided outside a tool: handing over to a person, asking for approval, or a guardrail blocking a reply (blocked) | middle lane, dashed |
Why order matters. “Was the claim read before the payout?” only makes sense when the steps are in order. That's why a session is a list, not a bag of facts.
Why calls and results are separate steps. Asking for a payout and the
payout succeeding are different things. A check like “don't say it's paid unless
it was” needs the result, not just the call. A result is matched to its call by
an id (call_id) when the agent records one, or else by tool name.
Turns and times. Each step also carries a turn number (turn 1
starts with the customer's first message, turn 2 with their second) and, where
the source records it, the time it happened (at). Traces record times;
chat messages don't.
Two ways today: from chat messages in OpenAI's format (how), or rebuilt from the agent's traces (how). Either way, the checks see the same shape.