cleandataengineering.aiEval Harness, drawn simply
You are here: Sessions

Eval Harness Tutorial · Sessions

What a session is

A session is one whole conversation between a customer and the agent, written down as a list of steps in the order they happened. It's the one thing every conversation-level check reads.

Here is a real one, in FNOL's words. A customer asks for a payout; the agent reads the claim, pays it, and says so:

CUSTOMERAGENTTOOLS (its systems)Customer“Please pay claim CLM-104233.”calls get_claimid=CLM-104233get_claim returnsid: CLM-104233, status: approved,approved_amount: 12400calls issue_payoutid=CLM-104233issue_payout returnspaid: TrueAgent says“The payout for CLM-104233 hasbeen issued.”
Six steps, top to bottom. Each box is one Step; its lane says who did it.

The five kinds of step

KindWhat it recordsDrawn as
userWhat the customer typedleft lane
assistantWhat the agent replied. A person's reply after a hand-over is one too, named human, and checks on the agent's own words skip it.middle lane
tool_callThe agent asked a system to do something: its name and argumentsright lane, dark
tool_resultWhat the system answered, or that it failedright lane, light
decisionSomething decided outside a tool: handing over to a person, asking for approval, or a guardrail blocking a reply (blocked)middle lane, dashed

Why order matters. “Was the claim read before the payout?” only makes sense when the steps are in order. That's why a session is a list, not a bag of facts.

Why calls and results are separate steps. Asking for a payout and the payout succeeding are different things. A check like “don't say it's paid unless it was” needs the result, not just the call. A result is matched to its call by an id (call_id) when the agent records one, or else by tool name.

Turns and times. Each step also carries a turn number (turn 1 starts with the customer's first message, turn 2 with their second) and, where the source records it, the time it happened (at). Traces record times; chat messages don't.

Where sessions come from

Two ways today: from chat messages in OpenAI's format (how), or rebuilt from the agent's traces (how). Either way, the checks see the same shape.

Look inside