Agent Harness Tutorial · Testing
Unit tests: table-driven, tied to the spec
Every behaviour is tested with a table: one row per case, with the boundary cases written out (one rupee under the limit, exactly on it, one rupee over). One test function runs every row. A marker on the test names the spec statements it proves, so a gate can later ask "is every promise proved by at least one passing test?".
Tests live one file per concern: test_payout_limit.py, test_policy.py, test_injection.py, test_redelivery.py, test_kill_switch.py, test_budgets.py, test_roles.py and so on. The model is a scripted client; the database and tools are in-memory adapters behind the same ports.
The ids come in three families: spec statements (P-*, R-*, op:*), assurance obligations (AAC-xxxx) and harness duties (AHC-xxxx). A second marker, unwired, marks a test of a part the agent never calls: its ids count for nothing.
How it is done: the real code
Cut from the live repositories each time the site is built, so it always matches what runs.
tests/test_payout_limit.py:50A real table-driven test: the payout limit's boundariesAMOUNTS = [
(1, False),
(8750, False),
(24999, False),
(25000, False), # exactly on the limit: at_most
(25001, True), # one rupee past it
(100000, True),
("not a number", True), # unreadable: a gate that cannot read the number never calls it small
(None, True),
]
@pytest.mark.discharges("P-PAYOUT", "P-APPROVER", "op:issue_payout", "AHC-0057")
@pytest.mark.parametrize(("amount", "needs_a_person"), AMOUNTS, ids=[str(a[0]) for a in AMOUNTS])
def test_the_limit_decides_who_authorises(amount: object, needs_a_person: bool) -> None:tests/conftest.py:1The @discharges plugin: each test's spec ids onto its JUnit record"""The discharges plugin: what each test verifies, on its JUnit record.
@pytest.mark.discharges("P-PAYOUT", "AHC-0057")
@pytest.mark.unwired # tests a component the agent never calls — counts for nothing
The gates (`clean-ai-engineering/tools/gates.py`) read `discharges` off the
JUnit report; an id that names nothing in the AOAS or the catalogs fails
collection before anything runs.
"""
from __future__ import annotations
import pytest
from evals import statements
_vocab: statements.Vocabulary | None = None
def pytest_configure(config: pytest.Config) -> None:
global _vocab
_vocab = statements.load()
def pytest_collection_modifyitems(config: pytest.Config, items: list[pytest.Item]) -> None:
bad: dict[str, list[str]] = {}
for item in items:
marker = item.get_closest_marker("discharges")
ids: list[str] = []
for mark in item.iter_markers("discharges"):
ids.extend(mark.args)
if marker is not None:
item.user_properties.append(("discharges", ",".join(dict.fromkeys(ids))))
if item.get_closest_marker("unwired") is not None:
item.user_properties.append(("unwired", "true"))