cleandataengineering.aiAgent Harness, drawn simply
You are here: Testing › Contract tests

Agent Harness Tutorial · Testing

Contract tests: every plug fits its socket

Each port (socket) has one table of contract rows. Every adapter registered for that port must pass the same rows. That is what makes "swap Groq for APIM" or "swap Keycloak for Entra" a configuration change instead of a rewrite.

modelthe same table of rowsscripted · litellm-proxy · groq-direct · apim-ai-gatewayidentitythe same table of rowslocal-dev · keycloak · entra-idsecretsthe same table of rowsenvironment-settings · key-vaulttelemetrythe same table of rowsconsole · otel-to-langfuse · azure-monitor-otelapprovalthe same table of rowstemporal-updates · dbos-workflowsrecordsthe same table of rowsin-memory · postgres
From the reference agent's tests/test_adapter_contracts.py.

The overlay that composes adapters is tested too: a wrong composition is refused at start-up, and every adapter must load and name its port. The Prometheus alert rules have their own unit tests, run with promtool. Planned: consumer contract tests for the MCP servers.

How it is done: the real code

Cut from the live repositories each time the site is built, so it always matches what runs.

reference-agent/tests/test_adapter_contracts.py:116The model port's contract rows, run against every model adapter
MODEL_ROWS: list[tuple[str, dict[str, Any], ModelResponse]] = [
    (
        "a text answer comes back typed, with its usage",
        completion("Your order has shipped."),
        ModelResponse(
            text="Your order has shipped.", usage=Usage(input_tokens=11, output_tokens=3)
        ),
    ),
    (
        "a tool call comes back as a ToolCall with parsed arguments",
        completion(None, [{"id": "c1", "name": "get_order", "arguments": {"id": "AB-10010"}}]),
        ModelResponse(
            tool_calls=(ToolCall(id="c1", name="get_order", arguments={"id": "AB-10010"}),),
            usage=Usage(input_tokens=11, output_tokens=3),
        ),
    ),
]
MODEL_ADAPTERS: list[tuple[str, dict[str, Any], str]] = [
    # (adapter, overlay settings, where the key must travel)
    ("scripted", {}, ""),
    ("litellm-proxy", {"api_key": "k-1", "base_url": "http://proxy.test/v1"}, "authorization"),
    ("groq-direct", {"api_key": "k-1", "base_url": "http://groq.test/v1"}, "authorization"),
    (
        "apim-ai-gateway",
        {"api_key": "k-1", "base_url": "http://apim.test/v1"},
        "ocp-apim-subscription-key",
    ),
]
ASK = ModelRequest(messages=(Message(role="user", content="Where is AB-10010?"),))


@pytest.mark.discharges("AHC-0022")
@pytest.mark.parametrize(
    ("name", "settings", "key_at"), MODEL_ADAPTERS, ids=[a[0] for a in MODEL_ADAPTERS]