Eval Harness Tutorial · Sessions
Building a session from chat messages
Almost every agent framework can hand you the conversation as
a list of messages in OpenAI's format. Session.from_messages turns
that list into steps, one message at a time.
Goes inA list of chat messages (user, assistant, tool)
What happensEach user message starts a new turn. An assistant message becomes a reply step, plus one tool_call step per tool it asked for. A tool message becomes a tool_result.
Comes outA Session with its steps in order
Goes in: four messages
[
{"role": "user", "content": "Please pay claim CLM-104233."},
{"role": "assistant", "tool_calls": [{"id": "c1", "function":
{"name": "get_claim", "arguments": "{\"id\": \"CLM-104233\"}"}}]},
{"role": "tool", "name": "get_claim", "tool_call_id": "c1",
"content": "{\"approved_amount\": 12400}"},
{"role": "assistant", "content": "Claim CLM-104233 is approved for ₹12,400."}
]Comes out: user, tool_call get_claim,
tool_result get_claim, assistant, all in turn 1.
The code
src/evalharness/core.py:194 def from_messages(
cls, id: str, messages: tuple[Mapping[str, Any], ...] | list[Mapping[str, Any]]
) -> Session:
"""From OpenAI-format chat messages, which most frameworks can export."""
import json
steps: list[Step] = []
turn = 0
for m in messages:
role = m.get("role")
if role == "user":
turn += 1
steps.append(Step("user", text=str(m.get("content") or ""), turn=turn))
elif role == "assistant":
if m.get("content"):
steps.append(Step("assistant", text=str(m["content"]), turn=turn))
for call in m.get("tool_calls") or ():
fn = call.get("function", {})
raw = fn.get("arguments") or "{}"
arguments = json.loads(raw) if isinstance(raw, str) else dict(raw)
steps.append(
Step(
"tool_call",
name=str(fn.get("name", "")),
arguments=arguments,
call_id=str(call.get("id", "")),
turn=turn,
)
)
elif role == "tool":
content = m.get("content")
try:
result: object = json.loads(content) if isinstance(content, str) else content
except ValueError:
result = content
steps.append(
Step(
"tool_result",
name=str(m.get("name", "")),
text=content if isinstance(content, str) else "",
result=result,
call_id=str(m.get("tool_call_id", "")),
is_error=bool(m.get("is_error", False)),
turn=turn,
)
)
return cls(id=id, steps=tuple(steps))What chat messages can't tell us: decisions (a hand-over is usually not a message) and whether a tool failed (most formats have no error flag). Traces carry more; see From traces.