Eval Harness Tutorial · Sessions
The Session in code
The whole shape is two small classes. Both are frozen (they can't be changed once built), so a check can never alter what another check will read.
Step and Session
src/evalharness/core.py:123StepKind = Literal["user", "assistant", "tool_call", "tool_result", "decision"]
HUMAN = "human"
"""The name of an `assistant` step a person wrote, after a hand-over."""
@dataclass(frozen=True)
class Step:
"""One thing that happened in a session, in order.
A `decision` is something the agent or the harness decided outside a tool
call, such as `escalate` or a guardrail's `blocked`; a host records it
under that name. A person's reply after a hand-over is an `assistant` step
named `human`: checks on the agent's own words skip it."""
kind: StepKind
name: str = ""
"""The tool's name for a call or result; the decision's name for a decision."""
text: str = ""
arguments: Mapping[str, object] = field(default_factory=dict)
result: object = None
"""A tool result's structured content."""
call_id: str = ""
"""Pairs a result with its call, where the host records it."""
is_error: bool = False
turn: int = 0
at: float | None = None
"""When it happened, in seconds since the epoch, where the source records
it (traces do; chat messages don't)."""
@property
def by_agent(self) -> bool:
"""An agent reply, not a person's after a hand-over."""
return self.kind == "assistant" and self.name != HUMAN
@dataclass(frozen=True)
class Session:
"""A whole conversation: what the session-level checks read."""
id: str
steps: tuple[Step, ...]
outcome: str = ""
"""How it ended, in the host's words: `resolved`, `abandoned`, `escalated`."""
unrecorded: frozenset[str] = frozenset()
"""What the source did not record, so a check must not read its absence as
a fact: `text` (what the customer and the agent said), `arguments` (tool
arguments), `results` (tool results). Production traces with content
capture off hold none of the three."""
def calls(self, name: str | None = None) -> tuple[tuple[int, Step], ...]:
"""Tool calls, with their position in `steps`, optionally one tool's only."""
return tuple(
(i, s)
for i, s in enumerate(self.steps)
if s.kind == "tool_call" and (name is None or s.name == name)
)
def succeeded(self, index: int) -> bool:
"""Whether the call at `index` has a result that is not an error. A
result is paired by call id where recorded, else the next one by name."""
call = self.steps[index]
for later in self.steps[index + 1 :]:
if later.kind != "tool_result":
continue
if (call.call_id and later.call_id == call.call_id) or (
not call.call_id and later.name == call.name
):
return not later.is_error
return FalseWhat a session does not hold yet
| Missing | Why it might matter | Today |
|---|---|---|
| Cost of each step | “This conversation cost ₹40 in model calls” | Not on the session; the cost engine will read calls separately |
| How it ended | resolved, handed_over, abandoned, error (decided 18 Oct) | The outcome field exists; standard traces have no attribute to fill it |
| Who the customer is | Per-customer limits (“3 refunds a month”) | Kept on the request's meta, never sent to a vendor |
Decided · The Session shape
- Are the five kinds of step enough? Is there anything an ops person asks about that isn't one of them (for example a guardrail blocking a reply, or a person replying after a hand-over)?
Decided: Keep five kinds. A guardrail blocking a reply is adecisionnamedblocked. A person's reply after a hand-over is anassistantstep namedhuman, which checks on the agent's own words skip. Built. - Should each step carry its time? It costs little, and enables “how long did the customer wait” rules.
Decided: Yes. Every step has an optional time (at); sessions rebuilt from traces fill it in. Built. - For
outcome: which words should a conversation end with in your world (resolved, abandoned, escalated, something else)?
Decided:resolved,handed_over,abandoned,error. A host may use its own words; reports group unknown ones as “other”.