cleandataengineering.aiAgent Harness, drawn simply
You are here: Testing › Unit tests

Agent Harness Tutorial · Testing

Unit tests: table-driven, tied to the spec

Every behaviour is tested with a table: one row per case, with the boundary cases written out (one rupee under the limit, exactly on it, one rupee over). One test function runs every row. A marker on the test names the spec statements it proves, so a gate can later ask "is every promise proved by at least one passing test?".

A table of casesamount → needs a person?24,999 · 25,000 · 25,001 ...One test function@parametrize over the table@discharges("P-PAYOUT", ...)Open: JUnit recordJUnit recordeach case + the spec idsit proveszoom in ›
The marker vocabulary is checked: a test naming a statement that doesn't exist fails collection.

Tests live one file per concern: test_payout_limit.py, test_policy.py, test_injection.py, test_redelivery.py, test_kill_switch.py, test_budgets.py, test_roles.py and so on. The model is a scripted client; the database and tools are in-memory adapters behind the same ports.

The ids come in three families: spec statements (P-*, R-*, op:*), assurance obligations (AAC-xxxx) and harness duties (AHC-xxxx). A second marker, unwired, marks a test of a part the agent never calls: its ids count for nothing.

How it is done: the real code

Cut from the live repositories each time the site is built, so it always matches what runs.

tests/test_payout_limit.py:50A real table-driven test: the payout limit's boundaries
AMOUNTS = [
    (1, False),
    (8750, False),
    (24999, False),
    (25000, False),  # exactly on the limit: at_most
    (25001, True),  # one rupee past it
    (100000, True),
    ("not a number", True),  # unreadable: a gate that cannot read the number never calls it small
    (None, True),
]


@pytest.mark.discharges("P-PAYOUT", "P-APPROVER", "op:issue_payout", "AHC-0057")
@pytest.mark.parametrize(("amount", "needs_a_person"), AMOUNTS, ids=[str(a[0]) for a in AMOUNTS])
def test_the_limit_decides_who_authorises(amount: object, needs_a_person: bool) -> None:
tests/conftest.py:1The @discharges plugin: each test's spec ids onto its JUnit record
"""The discharges plugin: what each test verifies, on its JUnit record.

    @pytest.mark.discharges("P-PAYOUT", "AHC-0057")
    @pytest.mark.unwired   # tests a component the agent never calls — counts for nothing

The gates (`clean-ai-engineering/tools/gates.py`) read `discharges` off the
JUnit report; an id that names nothing in the AOAS or the catalogs fails
collection before anything runs.
"""

from __future__ import annotations

import pytest
from evals import statements

_vocab: statements.Vocabulary | None = None


def pytest_configure(config: pytest.Config) -> None:
    global _vocab
    _vocab = statements.load()


def pytest_collection_modifyitems(config: pytest.Config, items: list[pytest.Item]) -> None:
    bad: dict[str, list[str]] = {}
    for item in items:
        marker = item.get_closest_marker("discharges")
        ids: list[str] = []
        for mark in item.iter_markers("discharges"):
            ids.extend(mark.args)
        if marker is not None:
            item.user_properties.append(("discharges", ",".join(dict.fromkeys(ids))))
        if item.get_closest_marker("unwired") is not None:
            item.user_properties.append(("unwired", "true"))