Drug-Safety Agent on Databricks · Core loop · The model
The model call
The loop talks to the model through one small port: send messages and the list of tools, get back words or tool requests. Because it is a port, the model is a setting. Tests use a scripted model, evaluation runs use a hosted open model, and Databricks' own Foundation Model APIs plug in the same way.
| Used for | Model | Where it ran |
|---|---|---|
| Unit tests and the worked run | A scripted model that returns fixed replies | In process; free and repeatable |
| The 20-pair evaluation (10 known signals, 10 controls) | gpt-oss-120b | Hosted by Groq's free tier. 20 of 20 finished as drafts, called both required tools, and passed the grounding check |
| Scenarios in the simulated world | A provider twin that answers over HTTP | AgentTwin, no cost |
| The Databricks build | Foundation Model APIs, for example databricks-gpt-oss-120b | Listed on the free workspace. Not yet run through the same 20-pair evaluation. |
Honest note. The recorded evaluation used the same open model but through another host, so it tells us about the model and the harness, not about Databricks' hosting. Re-running it against the Databricks endpoint is on the status list.
Which model to choose for which job is on the site's which model page.
The real code
Cut from the repository each time this site is built, so it always matches what runs. Each header names the file and the line it was cut from.
scripts/eval_run.py:18How the 20 pairs are chosen from the data, not from memorydef pick():
sig, ctl = [], []
for (d, e) in T._pairs():
s, l = T.signal_stats(d, e), T.label_mentions(d, e)
if s["screens_as_signal"] and l["on_label"]:
sig.append((s["prr"], d, e, "known_signal"))
elif not s["screens_as_signal"] and not l["on_label"]:
ctl.append((s["cases"], d, e, "control"))
sig = sorted(sig, reverse=True)[:10]
ctl = sorted(ctl, reverse=True)[:10] # controls with the most reports: hardest to dismiss as "no data"
return [(d, e, k) for _, d, e, k in sig + ctl]