Drug-safety signal investigator on Databricks
A pharmacovigilance agent built only on Databricks' own AI services, from public FDA data.
1 · Purpose
Drug-safety teams ask a recurring question: is there a signal that this drug causes this event? Answering it means counting adverse-event reports, comparing against other drugs, checking whether the label already warns about it, and judging whether the pattern is real or a reporting artefact. The path is not known in advance, which is what makes it a fit for an agent. This project builds that investigator end to end using Databricks' own AI offerings: Unity Catalog, Foundation Model APIs, Unity Catalog functions as tools, Mosaic AI Agent Framework and MLflow. It is also the worked example for the Databricks AI row of the platform page. A safety scientist approves every assessment; the agent files nothing. Data is public openFDA reports and drug labels.
2 · Business benefits
3 · Status and how to join
Good first tasks this week:
- Pick five drugs with a known label change for the test set
- Review the signal logic in the plan
4 · Requirements
What it must do
- Takes a drug and an event and reports how many cases exist and how that compares with other drugs.
- Checks whether the drug's label already mentions the event.
- Flags likely duplicate reports and obvious reporting artefacts.
- Drafts a signal assessment with the evidence and the uncertainty stated.
- Waits for a safety scientist to approve, edit or reject the assessment.
- Scores itself on drugs with known label changes and on control drugs.
How well it must do it
- The model calls named tools only; it never writes SQL.
- Statistics are deterministic and unit tested.
- Every run is traced and replayable.
- Cost per assessment is measured from system tables.
- Only public data is used; no employer data and no personal data.
- Anything a free workspace cannot do is recorded as a limit, not hidden.
5 · Architecture, very high level
6 · Tech stack, per platform
| Component | Databricks | Open stack |
|---|---|---|
| Public safety data | openFDA API, reachable ✓ | — |
| Governed tables | Delta tables in Unity Catalog | — |
| Statistics tools | Unity Catalog functions | — |
| Label finder | embeddings in Delta plus a function (Vector Search is not available on the free workspace) | — |
| Agent | Mosaic AI Agent Framework | not built |
| Model service | Foundation Model APIs, gpt-oss-120b and bge-large available ✓ | — |
| Tracing and evaluation | MLflow tracing and evaluation | — |
| Reviewer desk | Databricks Apps, or a job that writes a review file | — |
7 · What needs to be done
Phase 0: check 1 of 3 done
| Task | Status | Size | Skill |
|---|---|---|---|
| Check what Databricks AI services the free workspace allows (models, tables and Genie work; Vector Search, serving, warehouse start and model queries were refused on the day, cause unconfirmed) | doing | S | Databricks |
| Plan, mapping each need to a Databricks service | done | S | Architecture |
| Design against the agent harness, assurance and evaluation standards (Databricks stack file added; profile written) | doing | M | Architecture |
Phase 1: data 1 of 2 done
| Task | Status | Size | Skill |
|---|---|---|---|
| Ingest 10 drugs and 8 events: 21 million reports and labels, as local files | done | M | Data engineering |
| Load into Delta tables in Unity Catalog with data-quality checks | todo | M | Databricks |
Phase 2: tools and agent 2 of 4 done
| Task | Status | Size | Skill |
|---|---|---|---|
| Tools as plain Python with tests: counts, PRR and ROR, label check | done | M | SQL, Python |
| Register the tools as Unity Catalog functions | todo | M | SQL, Python |
| Agent loop with output contract, grounding check and 48 tests; run on a stand-in model | done | L | Agents |
| Package with Agent Framework and trace in MLflow | todo | L | Agents |
Phase 3: evaluation and guide 0 of 2 done
| Task | Status | Size | Skill |
|---|---|---|---|
| Score on 10 known signals and 10 controls | doing | M | AI testing |
| Write the guide, with the free-workspace limits stated | todo | M | Technical writing |
8 · Roles
| Role | How many | Hours / week | What they do |
|---|---|---|---|
| Databricks engineer | 1 | 3-5 hours/week | Owns the tables, functions and MLflow set-up. |
| Agent engineer | 1 | 3-5 hours/week | Builds the agent and its evaluation. |
| Drug-safety reviewer | 1 | 1-2 hours/week | Checks the signal logic and the assessment wording. |
9 · What you'll learn
- What each Databricks AI service does and when to use which.
- How to expose governed tools to an agent with Unity Catalog functions.
- How to trace and evaluate an agent with MLflow.
- How signal detection works: counts, comparisons and their limits.
Level-up questions
- Which steps of a signal assessment should never be left to the model?
- What changes when Vector Search and model serving are available?
- How would you cap the cost of one assessment?