cleandataengineering.aiEval Harness, drawn simply
You are here: Sessions › From chat messages

Eval Harness Tutorial · Sessions

Building a session from chat messages

Almost every agent framework can hand you the conversation as a list of messages in OpenAI's format. Session.from_messages turns that list into steps, one message at a time.

Goes inA list of chat messages (user, assistant, tool)
What happensEach user message starts a new turn. An assistant message becomes a reply step, plus one tool_call step per tool it asked for. A tool message becomes a tool_result.
Comes outA Session with its steps in order
Goes in: four messages
[
  {"role": "user", "content": "Please pay claim CLM-104233."},
  {"role": "assistant", "tool_calls": [{"id": "c1", "function":
      {"name": "get_claim", "arguments": "{\"id\": \"CLM-104233\"}"}}]},
  {"role": "tool", "name": "get_claim", "tool_call_id": "c1",
   "content": "{\"approved_amount\": 12400}"},
  {"role": "assistant", "content": "Claim CLM-104233 is approved for ₹12,400."}
]

Comes out: user, tool_call get_claim, tool_result get_claim, assistant, all in turn 1.

    def from_messages(
        cls, id: str, messages: tuple[Mapping[str, Any], ...] | list[Mapping[str, Any]]
    ) -> Session:
        """From OpenAI-format chat messages, which most frameworks can export."""
        import json

        steps: list[Step] = []
        turn = 0
        for m in messages:
            role = m.get("role")
            if role == "user":
                turn += 1
                steps.append(Step("user", text=str(m.get("content") or ""), turn=turn))
            elif role == "assistant":
                if m.get("content"):
                    steps.append(Step("assistant", text=str(m["content"]), turn=turn))
                for call in m.get("tool_calls") or ():
                    fn = call.get("function", {})
                    raw = fn.get("arguments") or "{}"
                    arguments = json.loads(raw) if isinstance(raw, str) else dict(raw)
                    steps.append(
                        Step(
                            "tool_call",
                            name=str(fn.get("name", "")),
                            arguments=arguments,
                            call_id=str(call.get("id", "")),
                            turn=turn,
                        )
                    )
            elif role == "tool":
                content = m.get("content")
                try:
                    result: object = json.loads(content) if isinstance(content, str) else content
                except ValueError:
                    result = content
                steps.append(
                    Step(
                        "tool_result",
                        name=str(m.get("name", "")),
                        text=content if isinstance(content, str) else "",
                        result=result,
                        call_id=str(m.get("tool_call_id", "")),
                        is_error=bool(m.get("is_error", False)),
                        turn=turn,
                    )
                )
        return cls(id=id, steps=tuple(steps))

What chat messages can't tell us: decisions (a hand-over is usually not a message) and whether a tool failed (most formats have no error flag). Traces carry more; see From traces.