Agent Harness Tutorial · Testing
The gates and CI
CI runs on every push. Static checks first, then the full test suite, then the four gates, then the container images, which are only pushed when everything is green on main.
What blocks. A merge is blocked by any failing check or test. A release is blocked by a gate failing, a reliability score (pass^k) under its threshold, or any successful attack. An uncertain verdict goes to a person and is never counted as a pass. The baseline may only go up.
pass^k means a case must pass all k repeated runs, not most of them: C(c, k) / C(n, k), averaged over scenarios. The gate runs scripted, once, today; live repeats are designed for nightly and weekly jobs, as are pre-commit hooks, live golden runs, judges and red team.
How it is done: the real code
Cut from the live repositories each time the site is built, so it always matches what runs.
.github/workflows/ci.yml:47.github/workflows/ci.ymljobs:
checks:
name: Checks, suite and gates
runs-on: ubuntu-24.04
timeout-minutes: 40
defaults:
run:
working-directory: claims-fnol-azure
services:
# PostgreSQL 16, as on the Mac and in Azure. No test needs `vector`, so the
# plain image. The service superuser has CREATEROLE, which the A4 role tests
# need (tests/pg.py makes and drops throwaway roles and databases).
postgres:
image: postgres:16
env:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: postgres
ports:
- 5432:5432
options: >-
--health-cmd "pg_isready -U postgres"
--health-interval 5s
--health-timeout 5s
--health-retries 10
env:
# tests/pg.py: the server, no database in the URL, a role that may create
# databases and roles. Nothing else is read from .env (there is none here).
# An IP, not `localhost`: psycopg resolves a host name on the event loop's
# default executor, which DBOS has shut down by the time the role
# teardown connects (FINDINGS F-103); an address needs no lookup.
CLAIMS_TEST_SERVER_URL: postgresql://postgres:postgres@127.0.0.1:5432gates.yaml:5gates.yamlapiVersion: gates/v0 agent: motor-claims-fnol aoas: ../clean-ai-engineering/drafts/examples/motor-claims-fnol.aoas.yaml profile: harness-profile.yaml # `agenttwin.Binding`: `(live, *, wrap, clock, model) -> Subject`. binding: evals.gate_binding:open_subject tests: # Each test carries its `discharges` ids as JUnit properties (tests/conftest.py). command: uv run python -m pytest -q -p no:cacheprovider --junitxml={junit} structure: checks: types: uv run mypy imports: uv run lint-imports complexity: uv run ruff check src tests evals # One module per harness layer (AHC taxonomy/layers.yaml). The engine of every # layer is the installed library, agent_harness — named where it lives, in the # reference agent's checkout this project depends on by path — and a layer # whose values are this agent's names its own module beside it. layers: L1: ../reference-agent/packages/agent-harness/src/agent_harness/context L2: ../reference-agent/packages/agent-harness/src/agent_harness/llm L3: ../reference-agent/packages/agent-harness/src/agent_harness/tools L4: - ../reference-agent/packages/agent-harness/src/agent_harness/loop - src/claims_fnol/router L5: ../reference-agent/packages/agent-harness/src/agent_harness/state L6: - ../reference-agent/packages/agent-harness/src/agent_harness/contracts - src/claims_fnol/contracts L7: - ../reference-agent/packages/agent-harness/src/agent_harness/policy
def pass_hat_k(passed: int, runs: int, k: int) -> float:
"""The chance that k attempts of this scenario all succeed.
`C(c, k) / C(n, k)`, which is 0 when fewer than k of the runs passed and 1
when all of them did. Asking for more attempts than were run is not a
number this can estimate, and says so by refusing rather than by returning
something plausible.
"""
if k < 1:
raise ValueError("k is a number of attempts, so at least one")
if k > runs:
raise ValueError(f"pass^{k} needs at least {k} runs; {runs} were made")
if passed < k:
return 0.0
return comb(passed, k) / comb(runs, k)