cleandataengineering.aiAgent Harness, drawn simply
You are here: End to end › Eval › Offline

Agent Harness Tutorial · End to end · Zoom level 2

Offline eval

Offline eval runs before every release, in CI. It plays whole conversations against a simulated world and checks what ended up true in that world. If anything is worse than before, the release is blocked.

Two boxes open their own pages: scenarios and the gate.

Recorded model calls (cassettes). A real model call is recorded once and then replayed, with no network and no cost. If the request changes (a different prompt, model or tool list), the replay refuses rather than silently calling the live model.

Red team. Attack cases are built in: generated injection cases, property tests, and a note planted in a claim. An attack is judged by the world afterwards: was anything paid, written or leaked? Any success blocks the release. Tools like promptfoo, PyRIT and garak are not built yet.

LLM judges may score offline, but only report until they have been calibrated against human labels. Only exact rules and calibrated judges may block.

How it is done: the real code

Cut from the live repositories each time the site is built, so it always matches what runs.

agent_harness/cassette/__init__.py:130A recorded model call is matched by a fingerprint of the request
def fingerprint(request: ModelRequest) -> str:
    """A stable identity for a request.

    Only what the provider is actually told: messages, tool names, and the
    sampling parameters that change the answer. Tool *schemas* are excluded —
    a description reworded between runs is not a different question, and
    including it would invalidate every cassette on a docstring edit.
    """
    material = {
        "messages": [
            {"role": m.role, "content": m.content, "tool_call_id": m.tool_call_id}
            for m in request.messages
        ],
        "tools": sorted(_tool_name(t) for t in request.tools),
        "max_tokens": request.max_tokens,
        # Unset reads as 0.0, what it meant before it could be unset (F-064),
        # so a recording made then is still the same question now.
        "temperature": 0.0 if request.temperature is None else request.temperature,
    }
    canonical = json.dumps(material, sort_keys=True, separators=(",", ":"), default=str)
    return hashlib.sha256(canonical.encode()).hexdigest()[:16]
jobs:
  checks:
    name: Checks, suite and gates
    runs-on: ubuntu-24.04
    timeout-minutes: 40
    defaults:
      run:
        working-directory: claims-fnol-azure

    services:
      # PostgreSQL 16, as on the Mac and in Azure. No test needs `vector`, so the
      # plain image. The service superuser has CREATEROLE, which the A4 role tests
      # need (tests/pg.py makes and drops throwaway roles and databases).
      postgres:
        image: postgres:16
        env:
          POSTGRES_USER: postgres
          POSTGRES_PASSWORD: postgres
          POSTGRES_DB: postgres
        ports:
          - 5432:5432
        options: >-
          --health-cmd "pg_isready -U postgres"
          --health-interval 5s
          --health-timeout 5s
          --health-retries 10

    env:
      # tests/pg.py: the server, no database in the URL, a role that may create
      # databases and roles. Nothing else is read from .env (there is none here).
      # An IP, not `localhost`: psycopg resolves a host name on the event loop's
      # default executor, which DBOS has shut down by the time the role
      # teardown connects (FINDINGS F-103); an address needs no lookup.
      CLAIMS_TEST_SERVER_URL: postgresql://postgres:postgres@127.0.0.1:5432

Tech choices (what we chose, Azure and open-source options): L9 · Determinism & replay

Look inside