cleandataengineering.aiAgent Harness, drawn simply
You are here: Testing › Cassettes

Agent Harness Tutorial · Testing

Recorded model calls (cassettes)

A cassette records real model calls once, at the model socket, and plays them back with no network and no cost. Tests and golden cases then run the real loop, deterministically, for free.

Recorderaround a real LLMClientsavesCassette filefingerprint → response+ model, tools, temploadsPlayeran LLMClientmatchthe responsemissCassetteMissnever a live call
Recorded at the LLMClient socket, not at HTTP, so retries and gateways don't matter.

The fingerprint is a hash of the messages, the sorted tool names, max_tokens and temperature. Tool descriptions are left out, so rewording one doesn't invalidate the cassette; adding a tool does. Replaying under a different model, tool list or temperature is refused as crossing a trust boundary: re-record instead.

A miss fails the run. The player has no fallback client, so a test can never quietly call a real model. To adopt: HTTP-level cassettes for tools (vcrpy).

How it is done: the real code

Cut from the live repositories each time the site is built, so it always matches what runs.

agent_harness/cassette/__init__.py:130The fingerprint a recorded call is matched by
def fingerprint(request: ModelRequest) -> str:
    """A stable identity for a request.

    Only what the provider is actually told: messages, tool names, and the
    sampling parameters that change the answer. Tool *schemas* are excluded —
    a description reworded between runs is not a different question, and
    including it would invalidate every cassette on a docstring edit.
    """
    material = {
        "messages": [
            {"role": m.role, "content": m.content, "tool_call_id": m.tool_call_id}
            for m in request.messages
        ],
        "tools": sorted(_tool_name(t) for t in request.tools),
        "max_tokens": request.max_tokens,
        # Unset reads as 0.0, what it meant before it could be unset (F-064),
        # so a recording made then is still the same question now.
        "temperature": 0.0 if request.temperature is None else request.temperature,
    }
    canonical = json.dumps(material, sort_keys=True, separators=(",", ":"), default=str)
    return hashlib.sha256(canonical.encode()).hexdigest()[:16]
class Recorder:
    """Wraps a live client and keeps what passed through.

    Transparent: the wrapped client sees the same requests it would have seen,
    and the caller gets the same responses. Recording never changes behaviour,
    which is the only way a recording is worth anything.
    """

    def __init__(self, inner: LLMClient, *, context: Context | str = "") -> None:
        self._inner = inner
        self.cassette = Cassette(context=context)

    async def complete(self, request: ModelRequest) -> ModelResponse:
        response = await self._inner.complete(request)
        self._check(response)
        self.cassette.exchanges.append(Exchange(fingerprint(request), response))
        return response

    def _check(self, response: ModelResponse) -> None:
        """A declared context that the provider contradicts is worse than none.

        The response says which model actually answered. If the recorder was
        told something else, every later replay would be validated against a
        claim that was false when it was written — so this fails at record time,
        where somebody is present to fix it.
        """
        if response.model and self.cassette.context and response.model not in self.cassette.context:
            raise TrustBoundaryCrossed(
                f"recording declared {self.cassette.context} but {response.model!r} answered"
            )