cleandataengineering.aiAgent Harness, drawn simply
You are here: Testing › Gates and CI

Agent Harness Tutorial · Testing

The gates and CI

CI runs on every push. Static checks first, then the full test suite, then the four gates, then the container images, which are only pushed when everything is green on main.

check out 6 reposuv syncruffmypylint-importsactionlintthe full suitethe four gatesupload verdictcompile Bicepbuild imagespush (main only)Behaviour29 of 29 scenariospassFeatures750 tests · 64/64 statementspassStructure16 of 16 layerspassHarness81 owed: 54 · 23 gaps · 4 n/apassTHE FOUR GATES · latest verdict
From .github/workflows/ci.yml and reports/gates/verdict.md in the claims agent.

What blocks. A merge is blocked by any failing check or test. A release is blocked by a gate failing, a reliability score (pass^k) under its threshold, or any successful attack. An uncertain verdict goes to a person and is never counted as a pass. The baseline may only go up.

pass^k means a case must pass all k repeated runs, not most of them: C(c, k) / C(n, k), averaged over scenarios. The gate runs scripted, once, today; live repeats are designed for nightly and weekly jobs, as are pre-commit hooks, live golden runs, judges and red team.

How it is done: the real code

Cut from the live repositories each time the site is built, so it always matches what runs.

.github/workflows/ci.yml:47.github/workflows/ci.yml
jobs:
  checks:
    name: Checks, suite and gates
    runs-on: ubuntu-24.04
    timeout-minutes: 40
    defaults:
      run:
        working-directory: claims-fnol-azure

    services:
      # PostgreSQL 16, as on the Mac and in Azure. No test needs `vector`, so the
      # plain image. The service superuser has CREATEROLE, which the A4 role tests
      # need (tests/pg.py makes and drops throwaway roles and databases).
      postgres:
        image: postgres:16
        env:
          POSTGRES_USER: postgres
          POSTGRES_PASSWORD: postgres
          POSTGRES_DB: postgres
        ports:
          - 5432:5432
        options: >-
          --health-cmd "pg_isready -U postgres"
          --health-interval 5s
          --health-timeout 5s
          --health-retries 10

    env:
      # tests/pg.py: the server, no database in the URL, a role that may create
      # databases and roles. Nothing else is read from .env (there is none here).
      # An IP, not `localhost`: psycopg resolves a host name on the event loop's
      # default executor, which DBOS has shut down by the time the role
      # teardown connects (FINDINGS F-103); an address needs no lookup.
      CLAIMS_TEST_SERVER_URL: postgresql://postgres:postgres@127.0.0.1:5432
gates.yaml:5gates.yaml
apiVersion: gates/v0
agent: motor-claims-fnol
aoas: ../clean-ai-engineering/drafts/examples/motor-claims-fnol.aoas.yaml
profile: harness-profile.yaml

# `agenttwin.Binding`: `(live, *, wrap, clock, model) -> Subject`.
binding: evals.gate_binding:open_subject

tests:
  # Each test carries its `discharges` ids as JUnit properties (tests/conftest.py).
  command: uv run python -m pytest -q -p no:cacheprovider --junitxml={junit}

structure:
  checks:
    types: uv run mypy
    imports: uv run lint-imports
    complexity: uv run ruff check src tests evals
  # One module per harness layer (AHC taxonomy/layers.yaml). The engine of every
  # layer is the installed library, agent_harness — named where it lives, in the
  # reference agent's checkout this project depends on by path — and a layer
  # whose values are this agent's names its own module beside it.
  layers:
    L1: ../reference-agent/packages/agent-harness/src/agent_harness/context
    L2: ../reference-agent/packages/agent-harness/src/agent_harness/llm
    L3: ../reference-agent/packages/agent-harness/src/agent_harness/tools
    L4:
      - ../reference-agent/packages/agent-harness/src/agent_harness/loop
      - src/claims_fnol/router
    L5: ../reference-agent/packages/agent-harness/src/agent_harness/state
    L6:
      - ../reference-agent/packages/agent-harness/src/agent_harness/contracts
      - src/claims_fnol/contracts
    L7:
      - ../reference-agent/packages/agent-harness/src/agent_harness/policy
def pass_hat_k(passed: int, runs: int, k: int) -> float:
    """The chance that k attempts of this scenario all succeed.

    `C(c, k) / C(n, k)`, which is 0 when fewer than k of the runs passed and 1
    when all of them did. Asking for more attempts than were run is not a
    number this can estimate, and says so by refusing rather than by returning
    something plausible.
    """
    if k < 1:
        raise ValueError("k is a number of attempts, so at least one")
    if k > runs:
        raise ValueError(f"pass^{k} needs at least {k} runs; {runs} were made")
    if passed < k:
        return 0.0
    return comb(passed, k) / comb(runs, k)