Agent Harness Tutorial · Testing
Contract tests: every plug fits its socket
Each port (socket) has one table of contract rows. Every adapter registered for that port must pass the same rows. That is what makes "swap Groq for APIM" or "swap Keycloak for Entra" a configuration change instead of a rewrite.
The overlay that composes adapters is tested too: a wrong composition is refused at start-up, and every adapter must load and name its port. The Prometheus alert rules have their own unit tests, run with promtool. Planned: consumer contract tests for the MCP servers.
How it is done: the real code
Cut from the live repositories each time the site is built, so it always matches what runs.
reference-agent/tests/test_adapter_contracts.py:116The model port's contract rows, run against every model adapterMODEL_ROWS: list[tuple[str, dict[str, Any], ModelResponse]] = [
(
"a text answer comes back typed, with its usage",
completion("Your order has shipped."),
ModelResponse(
text="Your order has shipped.", usage=Usage(input_tokens=11, output_tokens=3)
),
),
(
"a tool call comes back as a ToolCall with parsed arguments",
completion(None, [{"id": "c1", "name": "get_order", "arguments": {"id": "AB-10010"}}]),
ModelResponse(
tool_calls=(ToolCall(id="c1", name="get_order", arguments={"id": "AB-10010"}),),
usage=Usage(input_tokens=11, output_tokens=3),
),
),
]
MODEL_ADAPTERS: list[tuple[str, dict[str, Any], str]] = [
# (adapter, overlay settings, where the key must travel)
("scripted", {}, ""),
("litellm-proxy", {"api_key": "k-1", "base_url": "http://proxy.test/v1"}, "authorization"),
("groq-direct", {"api_key": "k-1", "base_url": "http://groq.test/v1"}, "authorization"),
(
"apim-ai-gateway",
{"api_key": "k-1", "base_url": "http://apim.test/v1"},
"ocp-apim-subscription-key",
),
]
ASK = ModelRequest(messages=(Message(role="user", content="Where is AB-10010?"),))
@pytest.mark.discharges("AHC-0022")
@pytest.mark.parametrize(
("name", "settings", "key_at"), MODEL_ADAPTERS, ids=[a[0] for a in MODEL_ADAPTERS]