cleandataengineering.aiDrug-Safety Agent on Databricks, drawn simply
You are here: Eval and gates › Table-driven tests

Drug-Safety Agent on Databricks · Eval and gates · Table-driven tests

Table-driven tests

The tests are written as tables. A test function walks a list of rows, each row a request and the answer it must get. This keeps tests short, makes a missing case obvious, and lets a non-programmer read what is promised.

Tests carry the ids they prove. Each test is tagged with the specification ids it discharges, such as R-CAUSALITY. The gates read those tags to know which promises have evidence.

The loop is tested without a real model. A scripted model returns fixed replies, so a test can say "the model asks for two tools, then replies with this JSON" and check the draft, the steps and the spend.

The real code

Cut from the repository each time this site is built, so it always matches what runs. Each header names the file and the line it was cut from.

tests/test_router.py:35One function, every row of the table
@pytest.mark.discharges("R-CAUSALITY", "R-SUBMISSION", "R-ADVICE", "P-DIRECT", "P-DIRECT-READS", "esc:asked-for-human", "AHC-0100")
@pytest.mark.parametrize("text,intent,by,refusal,drug,event", ROUTES, ids=[r[0][:48] for r in ROUTES])
def test_route(text, intent, by, refusal, drug, event):
    r = route(text)
    assert (r.intent, r.answered_by, r.refusal, r.drug, r.event) == (intent, by, refusal, drug, event)
tests/test_hagent.py:40The loop with a scripted model
@pytest.mark.discharges("P-NUMBERS-FROM-TOOLS", "Q-GROUNDED", "P-OUTPUT-CONTRACT", "P-MODEL-ERROR")
@pytest.mark.asyncio
@pytest.mark.parametrize("name,script,status,problems,tools", CASES, ids=[c[0] for c in CASES])
async def test_investigate(name, script, status, problems, tools):
    d = await investigate(ScriptedClient(script), "amiodarone", "PULMONARY TOXICITY")
    assert d.status == status
    assert [t["name"] for t in d.trace] == tools
    found = grounding(d)
    assert len(found) == len(problems)
    for want in problems:
        assert any(want in f for f in found)


@pytest.mark.discharges("Q-STEPS", "AHC-0041")
@pytest.mark.asyncio
async def test_a_model_that_never_finishes_is_stopped_by_the_library_step_budget():
    d = await investigate(ScriptedClient([call("signal_stats", "1")] * 20), "amiodarone", "PULMONARY TOXICITY")
    assert d.status == "needs_human"
    assert d.steps <= BUDGETS.max_steps
tests/conftest.py:1How a test says which promise it proves
"""The discharges plugin: what each test verifies, on its JUnit record.

    @pytest.mark.discharges("P-PAYOUT", "AHC-0057")
    @pytest.mark.unwired   # tests a component the agent never calls — counts for nothing

The gates (`clean-ai-engineering/tools/gates.py`) read `discharges` off the
JUnit report; an id that names nothing in the AOAS or the catalogs fails
collection before anything runs.
"""

from __future__ import annotations

import pytest
from evals import statements