Agent Harness Tutorial · End to end · Zoom level 2
Offline eval
Offline eval runs before every release, in CI. It plays whole conversations against a simulated world and checks what ended up true in that world. If anything is worse than before, the release is blocked.
Recorded model calls (cassettes). A real model call is recorded once and then replayed, with no network and no cost. If the request changes (a different prompt, model or tool list), the replay refuses rather than silently calling the live model.
Red team. Attack cases are built in: generated injection cases, property tests, and a note planted in a claim. An attack is judged by the world afterwards: was anything paid, written or leaked? Any success blocks the release. Tools like promptfoo, PyRIT and garak are not built yet.
LLM judges may score offline, but only report until they have been calibrated against human labels. Only exact rules and calibrated judges may block.
How it is done: the real code
Cut from the live repositories each time the site is built, so it always matches what runs.
agent_harness/cassette/__init__.py:130A recorded model call is matched by a fingerprint of the requestdef fingerprint(request: ModelRequest) -> str:
"""A stable identity for a request.
Only what the provider is actually told: messages, tool names, and the
sampling parameters that change the answer. Tool *schemas* are excluded —
a description reworded between runs is not a different question, and
including it would invalidate every cassette on a docstring edit.
"""
material = {
"messages": [
{"role": m.role, "content": m.content, "tool_call_id": m.tool_call_id}
for m in request.messages
],
"tools": sorted(_tool_name(t) for t in request.tools),
"max_tokens": request.max_tokens,
# Unset reads as 0.0, what it meant before it could be unset (F-064),
# so a recording made then is still the same question now.
"temperature": 0.0 if request.temperature is None else request.temperature,
}
canonical = json.dumps(material, sort_keys=True, separators=(",", ":"), default=str)
return hashlib.sha256(canonical.encode()).hexdigest()[:16].github/workflows/ci.yml:47CI on every pushjobs:
checks:
name: Checks, suite and gates
runs-on: ubuntu-24.04
timeout-minutes: 40
defaults:
run:
working-directory: claims-fnol-azure
services:
# PostgreSQL 16, as on the Mac and in Azure. No test needs `vector`, so the
# plain image. The service superuser has CREATEROLE, which the A4 role tests
# need (tests/pg.py makes and drops throwaway roles and databases).
postgres:
image: postgres:16
env:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: postgres
ports:
- 5432:5432
options: >-
--health-cmd "pg_isready -U postgres"
--health-interval 5s
--health-timeout 5s
--health-retries 10
env:
# tests/pg.py: the server, no database in the URL, a role that may create
# databases and roles. Nothing else is read from .env (there is none here).
# An IP, not `localhost`: psycopg resolves a host name on the event loop's
# default executor, which DBOS has shut down by the time the role
# teardown connects (FINDINGS F-103); an address needs no lookup.
CLAIMS_TEST_SERVER_URL: postgresql://postgres:postgres@127.0.0.1:5432Tech choices (what we chose, Azure and open-source options): L9 · Determinism & replay