cleandataengineering.aiEval Harness, drawn simply
You are here: Sessions › In code

Eval Harness Tutorial · Sessions

The Session in code

The whole shape is two small classes. Both are frozen (they can't be changed once built), so a check can never alter what another check will read.

Sessionid: strsteps: tuple[Step, ...]outcome: strcalls(name) → the tool callssucceeded(i) → did call i work?from_messages(id, messages)has many, in orderStepkind: user · assistant · tool_call · tool_result · decisionname: str (tool or decision)text: strarguments: dictresult: objectcall_id: stris_error: bool · turn: intat: float | None (when)EvalRequestsession: Session(what every checkis handed)
Session holds Steps in order. A check is handed an EvalRequest, which carries the Session.
StepKind = Literal["user", "assistant", "tool_call", "tool_result", "decision"]
HUMAN = "human"
"""The name of an `assistant` step a person wrote, after a hand-over."""


@dataclass(frozen=True)
class Step:
    """One thing that happened in a session, in order.

    A `decision` is something the agent or the harness decided outside a tool
    call, such as `escalate` or a guardrail's `blocked`; a host records it
    under that name. A person's reply after a hand-over is an `assistant` step
    named `human`: checks on the agent's own words skip it."""

    kind: StepKind
    name: str = ""
    """The tool's name for a call or result; the decision's name for a decision."""
    text: str = ""
    arguments: Mapping[str, object] = field(default_factory=dict)
    result: object = None
    """A tool result's structured content."""
    call_id: str = ""
    """Pairs a result with its call, where the host records it."""
    is_error: bool = False
    turn: int = 0
    at: float | None = None
    """When it happened, in seconds since the epoch, where the source records
    it (traces do; chat messages don't)."""

    @property
    def by_agent(self) -> bool:
        """An agent reply, not a person's after a hand-over."""
        return self.kind == "assistant" and self.name != HUMAN


@dataclass(frozen=True)
class Session:
    """A whole conversation: what the session-level checks read."""

    id: str
    steps: tuple[Step, ...]
    outcome: str = ""
    """How it ended, in the host's words: `resolved`, `abandoned`, `escalated`."""
    unrecorded: frozenset[str] = frozenset()
    """What the source did not record, so a check must not read its absence as
    a fact: `text` (what the customer and the agent said), `arguments` (tool
    arguments), `results` (tool results). Production traces with content
    capture off hold none of the three."""

    def calls(self, name: str | None = None) -> tuple[tuple[int, Step], ...]:
        """Tool calls, with their position in `steps`, optionally one tool's only."""
        return tuple(
            (i, s)
            for i, s in enumerate(self.steps)
            if s.kind == "tool_call" and (name is None or s.name == name)
        )

    def succeeded(self, index: int) -> bool:
        """Whether the call at `index` has a result that is not an error. A
        result is paired by call id where recorded, else the next one by name."""
        call = self.steps[index]
        for later in self.steps[index + 1 :]:
            if later.kind != "tool_result":
                continue
            if (call.call_id and later.call_id == call.call_id) or (
                not call.call_id and later.name == call.name
            ):
                return not later.is_error
        return False

What a session does not hold yet

MissingWhy it might matterToday
Cost of each step“This conversation cost ₹40 in model calls”Not on the session; the cost engine will read calls separately
How it endedresolved, handed_over, abandoned, error (decided 18 Oct)The outcome field exists; standard traces have no attribute to fill it
Who the customer isPer-customer limits (“3 refunds a month”)Kept on the request's meta, never sent to a vendor
Decided · The Session shape
  1. Are the five kinds of step enough? Is there anything an ops person asks about that isn't one of them (for example a guardrail blocking a reply, or a person replying after a hand-over)?
    Decided: Keep five kinds. A guardrail blocking a reply is a decision named blocked. A person's reply after a hand-over is an assistant step named human, which checks on the agent's own words skip. Built.
  2. Should each step carry its time? It costs little, and enables “how long did the customer wait” rules.
    Decided: Yes. Every step has an optional time (at); sessions rebuilt from traces fill it in. Built.
  3. For outcome: which words should a conversation end with in your world (resolved, abandoned, escalated, something else)?
    Decided: resolved, handed_over, abandoned, error. A host may use its own words; reports group unknown ones as “other”.