cleandataengineering.aiDrug-Safety Agent on Databricks, drawn simply
You are here: Core loop › The model

Drug-Safety Agent on Databricks · Core loop · The model

The model call

The loop talks to the model through one small port: send messages and the list of tools, get back words or tool requests. Because it is a port, the model is a setting. Tests use a scripted model, evaluation runs use a hosted open model, and Databricks' own Foundation Model APIs plug in the same way.

Used forModelWhere it ran
Unit tests and the worked runA scripted model that returns fixed repliesIn process; free and repeatable
The 20-pair evaluation (10 known signals, 10 controls)gpt-oss-120bHosted by Groq's free tier. 20 of 20 finished as drafts, called both required tools, and passed the grounding check
Scenarios in the simulated worldA provider twin that answers over HTTPAgentTwin, no cost
The Databricks buildFoundation Model APIs, for example databricks-gpt-oss-120bListed on the free workspace. Not yet run through the same 20-pair evaluation.

Honest note. The recorded evaluation used the same open model but through another host, so it tells us about the model and the harness, not about Databricks' hosting. Re-running it against the Databricks endpoint is on the status list.

Which model to choose for which job is on the site's which model page.

The real code

Cut from the repository each time this site is built, so it always matches what runs. Each header names the file and the line it was cut from.

scripts/eval_run.py:18How the 20 pairs are chosen from the data, not from memory
def pick():
    sig, ctl = [], []
    for (d, e) in T._pairs():
        s, l = T.signal_stats(d, e), T.label_mentions(d, e)
        if s["screens_as_signal"] and l["on_label"]:
            sig.append((s["prr"], d, e, "known_signal"))
        elif not s["screens_as_signal"] and not l["on_label"]:
            ctl.append((s["cases"], d, e, "control"))
    sig = sorted(sig, reverse=True)[:10]
    ctl = sorted(ctl, reverse=True)[:10]  # controls with the most reports: hardest to dismiss as "no data"
    return [(d, e, k) for _, d, e, k in sig + ctl]