Home › On your cloud › Databricks › Build an agent
Databricks · AI
Build an agent on Databricks: the whole design, end to end
The purpose is two things at once: build a real agent on Databricks, and learn Databricks' AI offerings by using each one for a job. The agent is a drug-safety signal investigator: it counts adverse-event reports, computes statistics in code, checks the drug's label, and drafts an assessment for a person to decide. It follows the same seven parts as the claims (FNOL) guide.
Status as of 19 November 2026. The code is not public yet. Where this page says "works", we ran it on a Databricks Free Edition workspace.
What we use, what it is for, and how
| Service | What it is for | How this agent uses it | Status |
|---|---|---|---|
| Unity Catalog | One place that names, secures and tracks data, files, functions and models. | Holds the schema, the tables, the raw-file volume and the agent's tools. The agent's login is granted only those tools and tables. | works |
| Delta Lake tables | Tables that are safe to update and keep their history. | 80 drug-and-event count rows and 10 drug labels, loaded from public FDA data. | works |
| SQL warehouse (serverless) | Compute that runs SQL and registered functions. | Runs the table loads and the tool functions. | works |
| Unity Catalog functions | Small named functions, registered and permissioned, that an agent may call as tools. | Three read-only tools: report counts, signal statistics, label check. The model can call these and nothing else, so it never writes SQL. | works |
| Foundation Model APIs | Ready-hosted models you pay for per use, behind an OpenAI-style web address. | The reasoning model (gpt-oss-120b) ran the agent loop with tool calls and wrote a draft assessment. | works |
| MLflow tracing and evaluation | Records every step of a run, and scores answers. | The recorder: each model call and tool call becomes a trace the eval harness reads. Also scores the 20-case test set. | planned |
| Vector Search | Finds text by meaning, with an index that stays in step with a Delta table. | Finding the label passage for an adverse event, so 'swelling of the face' can find 'angioedema'. A keyword-and-synonym lookup does the job meanwhile. | planned |
| Mosaic AI Agent Framework | Databricks' standard way to package, register and deploy an agent so it can be traced and served. | A wrapper around our own loop (the loop stays ours); gives it the standard interface, tracing and registration. | planned |
| Databricks Apps | Small web apps that run inside the workspace, signed in as the user. | The screen where the safety scientist approves, edits or rejects a draft. | planned |
| Lakeflow Jobs | Scheduled and triggered runs with retries. | Runs an investigation as a job; a wait for a person ends the run and a new run starts on the decision. | planned |
| System tables | Databricks' own usage, billing and audit records as tables. | Cost per assessment from real billing rows. | planned |
| AI Gateway (default settings only) | A control point in front of model endpoints: usage tracking, rate limits, guardrails, fallbacks. | Usage tracking is on for the hosted model endpoint by default, so every call is counted by the platform. We configure nothing else on it. | works |
| Genie | Ask a governed table a question in plain English. | A comparison, not part of the agent: what can Genie answer about the counts alone, and where does it stop? | planned |
What we are not using, and why
| Service | What it is for | Why we do not use it here |
|---|---|---|
| Agent Bricks | Higher-level builders for common agents (question answering over documents, information extraction, a supervisor over several agents), mostly without code. | It hides the parts this build exists to learn, and our agent has a typed tool contract, an approval step and its own checks. Worth knowing for a customer who wants a quick answer over documents. |
| AI Functions (ai_query, ai_classify and similar) | SQL functions that call a model once per row. | Our agent decides which tools to call per case. It is not a per-row batch job. They would fit tagging thousands of report narratives. |
| Custom Model Serving | Host your own model or agent as a web service. | A person reviews each investigation later, so a job is enough and a live endpoint adds cost and a way in. Revisit when the agent is used interactively. |
| AI Gateway: limits, guardrails and fallbacks | A control point in front of model endpoints. | The hosted model endpoint we call already has AI Gateway attached with usage tracking on, by default. We have not configured rate limits, guardrails, request logging or fallbacks: our own harness enforces the step limit, the output contract and the grounding check. Worth configuring before more than one team uses the same endpoint. |
| Feature engineering, model training and fine-tuning | Build and tune your own models. | There is no model to train: the statistics are formulas and the language model is used as it is. |
| Provisioned throughput | Reserved model capacity. | Pay-per-use is enough at this volume. |
| Lakehouse Monitoring | Profiles data and tracks drift in tables. | Optional here; the eval harness watches the agent, not the tables. |
| Delta Sharing, Clean Rooms, Marketplace | Share data with other organisations. | Nothing is shared. |
1 · Requirements
What it must do
- Take a drug and an adverse event, and report the counts of FDA reports.
- Compute disproportionality statistics (PRR, ROR) in code.
- Check whether the drug's current label already mentions the event.
- Draft an assessment, with its uncertainty stated, for a person to approve, edit or reject.
How well
- Every number and label statement in a draft equals what a tool returned, in 100% of drafts.
- At most 6 steps and about USD 0.05 of model spend per investigation.
- The agent drafts and never files, sends or decides.
- Text inside a label is data, never an instruction.
2 · Architecture
The idea
- The same agent harness as the other agents on this site: a loop, tools behind a contract, policy, cost limits, approval, tracing. Only the bindings change between clouds.
- On Databricks, each port of the harness is filled by a Databricks service: the model by Foundation Model APIs, the tools by Unity Catalog functions, the state by Delta tables, the telemetry by MLflow, the identity by a service principal, the approval screen by an App.
3 · Design
Security
- Least privilege: the agent's login can run exactly its three functions and read exactly its tables.
- No SQL from the model; typed functions only.
- Label text is untrusted: quotes enter the context as data. A hostile instruction hidden in a quote is tested and changes nothing.
Reliability
- The final answer has a typed output contract. An invalid answer gets one repair turn, then goes to a person.
- A failed model call is retried once, then goes to a person with the reason.
- A rate limit makes the client wait as long as the server asks.
Cost
- Cost is measured per investigation, from tokens now and from billing system tables later.
- Statistics are code, so the model never spends tokens on arithmetic.
4 · Development
What exists
- Data pipeline from the public FDA API into Delta tables.
- Three tools, first as tested Python and now as Unity Catalog functions, which must give the same answers.
- The agent loop, the output contract and a grounding check.
- The agent running on the shared harness loop, with a Databricks model, end to end: three tool calls, a clean grounding check, goal reached in four steps.
- The specification (AOAS), a generated brief of 76 capabilities and 57 obligations, and the first Databricks adapter in the shared harness library.
Honest note
- The first version was written standalone, outside the shared harness library, to test feasibility. It has since been rebuilt on the library, the way the other agents are built: the loop, the step and cost limits and the model port are the library's, and only the tools, the prompt, the output contract and the grounding check are this agent's. Running it on the Databricks model found a real incompatibility (Databricks rejects the newer output-cap field the library sent); it was fixed in the library's Databricks adapter and is covered by a test.
5 · Testing
What ran
- 48 unit tests, table-driven, on the tools, the statistics and the loop (model invents a number, skips a tool, calls a tool that does not exist, never finishes, sends bad JSON, hides an instruction in a label).
- A 20-case run (10 known signals, 10 controls, ground truth taken from the data and the labels) on a stand-in model: 20 of 20 drafts, both required tools called, every number matched the tools.
- One run on the Databricks model: three tools, clean grounding check, 31 seconds.
Still to do
- Run the 20 cases on the Databricks model.
- Judge the quality of the written assessments, calibrated against human labels.
- A simulated world and scenarios in AgentTwin, and the four gates.
6 · Deploy and release
Plan
- An Asset Bundle that creates the schema, tables, functions and job together.
- The agent runs as a job under a service principal; releases are bundle versions.
- Vector Search and the approval App are added once their tiers pass.
7 · Monitor
Plan
- MLflow traces are read by the eval harness, which answers 62 everyday questions about any agent (is it failing, how fast, what does it cost, which tools fail, which guardrails fire) on a dashboard.
- The same 62 questions run on the Azure version of the harness, so one set of checks and questions covers both clouds. Where a platform cannot answer one, the dashboard says so and why.
Start here: the order to learn and build it
- Governed data. Create a schema, load a few tables, grant access. Draw the 2x2 table for one drug and one event by hand.
- Tools. Write one function that takes a drug and an event and returns the counts. Test it against a plain-Python answer.
- Model and loop. Call a hosted model with the function listed as a tool. Read one run's steps from start to finish.
- Checks. Add an output contract and a grounding check (every number equals a tool result). Break them on purpose with a scripted model.
- Tracing and evaluation. Trace a run in MLflow, then score 20 cases with known answers.
- People and release. Add the approval screen, run it as a job, and measure cost per investigation.
Each service explained
The same services, one at a time, in the order a request meets them. Names and limits change often; check the current Databricks docs before you build.
1 · Governed data
Unity Catalog
One place that names, secures and tracks everything: tables, files, functions and models.
- Problem it solves
- Without it, every tool has its own login and nobody can say who touched what.
- How it works
- Every object has a three-part name, catalog.schema.object. Permissions are granted on those names, and lineage records what read or wrote what.
- Small example
GRANT SELECT ON TABLE workspace.drugsafety.pair_counts TO `agent-service-principal`- In this agent
- Holds the schema, the tables, a volume for the raw files and the three tool functions. The agent's login is granted only those.
- Usual confusion
- It is not a database. It is the permission and lineage layer over data stored elsewhere.
Delta Lake tables
A table format that is safe to update and remembers its history.
- Problem it solves
- Plain files cannot be updated safely by two writers, and cannot answer 'what did this look like last week?'.
- How it works
- Data files plus a transaction log. Every change is a new version; older versions stay readable.
- Small example
SELECT * FROM pair_counts VERSION AS OF 3- In this agent
- Holds 80 drug-and-event count rows and 10 drug labels from public FDA data. Later: each draft assessment and each reviewer decision, so an audit can replay the past.
- Usual confusion
- A Delta table is not a different database. It is files in cloud storage with a log beside them.
SQL warehouse (serverless)
Compute that runs SQL and the registered functions.
- Problem it solves
- Something has to run the queries and functions; you do not want to manage a cluster to do it.
- How it works
- A serverless warehouse starts on demand and stops when idle. The Statement Execution API runs one SQL statement and returns rows.
- Small example
POST /api/2.0/sql/statements with a warehouse id and a statement- In this agent
- Runs the table loads, the function registration and every tool call.
- Usual confusion
- A warehouse is for SQL. Notebooks and jobs use other compute.
2 · Models
Foundation Model APIs
Ready-hosted models you call and pay for per use, with no servers to run.
- Problem it solves
- Hosting a large model yourself is slow and costly.
- How it works
- Each hosted model is an endpoint with an OpenAI-style chat format, including tool calls. Open models (gpt-oss, Llama, Qwen, Gemma) and embedding models (bge, gte) are listed in the workspace.
- Small example
POST /serving-endpoints/databricks-gpt-oss-120b/invocations with a list of chat messages- In this agent
- The reasoning model runs the loop: it chooses which tool to call, reads the results and drafts the assessment.
- Usual confusion
- Pay-per-token and provisioned throughput are two ways of buying the same models: share a pool, or reserve capacity.
Model Serving and AI Gateway
Host your own model or agent as a web service; AI Gateway puts limits, guardrails and logging in front of model endpoints.
- Problem it solves
- Without a control point, every team enforces its own rate limit, privacy rule and cost report.
- How it works
- Serving turns a registered model or agent into an endpoint that scales up and down. AI Gateway adds rate limits, guardrails, usage tracking and fallbacks in front of endpoints.
- Small example
an endpoint with a rate limit of N calls per minute per user- In this agent
- Custom Model Serving is not used: each investigation is reviewed by a person afterwards, so a job is enough. AI Gateway is attached to the hosted model endpoint with usage tracking on by default (we checked the endpoint); we have not configured rate limits, guardrails or fallbacks.
- Usual confusion
- Foundation Model APIs are hosted for you. Model Serving is where you host your own.
3 · Tools and knowledge
Unity Catalog functions as tools
A small named SQL or Python function, registered in Unity Catalog, that an agent may call.
- Problem it solves
- Letting a model write SQL is unsafe: it can read anything its login can, and text in the data can talk it into doing so.
- How it works
- A function has typed inputs, a comment the model reads as its description, a permission and an audit trail. The agent is shown only the functions registered for it.
- Small example
CREATE FUNCTION workspace.drugsafety.report_counts(drug STRING, event STRING) RETURNS TABLE (...) RETURN SELECT ... WHERE t.drug = lower(report_counts.drug) AND t.event = upper(report_counts.event)- In this agent
- The three tools: report counts, signal statistics (PRR, ROR and the screen, computed in SQL) and label check. A test runs all 80 drug-and-event pairs through the functions and through the tested Python versions: every answer matches.
- Usual confusion
- A tool is a contract. The model chooses a tool and its inputs, and the platform decides whether it may run.
Vector Search
Finds text by meaning, with an index that stays in step with a Delta table.
- Problem it solves
- Keyword search misses 'swelling of the face' when the label says 'angioedema'.
- How it works
- An embedding model turns text into a vector, a list of numbers where similar meanings sit close together. A query is embedded the same way and the nearest vectors win. A Delta Sync index re-embeds when the source table changes and inherits its permissions.
- Small example
search the labels for 'swelling of the face' and get the angioedema passage- In this agent
- Planned for finding the label passage about an event. Today a synonym table does the job, and the guide's next step swaps it.
- Usual confusion
- A vector index finds similar text, not true statements. A retrieved passage still has to be checked.
AI Functions
SQL functions that call a model once per row: ai_query, ai_classify, ai_extract and others.
- Problem it solves
- Tagging or extracting from thousands of rows should not need a loop in Python.
- How it works
- You call the function inside a SELECT, and the warehouse sends each row to a model endpoint.
- Small example
SELECT ai_classify(narrative, ARRAY('serious', 'non-serious')) FROM reports- In this agent
- Not used. The agent chooses its steps per case; it is not a per-row batch job. They would fit tagging thousands of report narratives.
- Usual confusion
- They are not an agent. They run the same model call on every row.
Genie
Ask a governed table a question in plain English and get SQL and an answer.
- Problem it solves
- Many people need an answer from a table and cannot write SQL.
- How it works
- A Genie space is configured with tables, instructions and example questions; it writes SQL, runs it and shows the result.
- Small example
'How many reports of angioedema for lisinopril?' returns the count and the SQL behind it- In this agent
- A comparison, not part of the agent: what Genie answers about the counts alone, and where it stops. It cannot weigh evidence or draft an assessment.
- Usual confusion
- Genie answers questions about data. It does not carry out a multi-step investigation with approval.
4 · The agent
Mosaic AI Agent Framework and Agent Bricks
Agent Framework is Databricks' way to write, log, test, register and deploy an agent. Agent Bricks builds common agents for you, mostly without code.
- Problem it solves
- Agents need a standard shape to be traced, evaluated and served consistently.
- How it works
- With Agent Framework you write the agent as a class with a standard interface (ResponsesAgent in current MLflow); MLflow logs it, Unity Catalog registers it, and a deploy step puts it behind an endpoint with a review app. Agent Bricks skips the code for document question-answering, information extraction and multi-agent supervision.
- Small example
log the agent with MLflow, register it as workspace.drugsafety.signal_investigator, deploy it- In this agent
- Agent Framework is planned as a wrapper around our own loop, for tracing and registration. Agent Bricks is not used: this build exists to see and control the parts.
- Usual confusion
- Agent Framework is not a loop engine like LangGraph. It standardises packaging, tracing and deployment.
Databricks Apps
Run a small web app (Streamlit, Dash, Gradio or FastAPI) inside the workspace, signed in as the user.
- Problem it solves
- You need a screen for people to act on what the agent produced, without hosting a server elsewhere.
- How it works
- You deploy app code; the platform runs it, handles sign-in and lets it query governed data on the user's behalf.
- Small example
a page listing drafts, with Approve, Edit and Reject buttons that write a row- In this agent
- Planned for the reviewer's screen: the decision is saved as a row in a Delta table.
- Usual confusion
- An App is a front end. The agent still runs as a job.
Lakeflow Jobs
Scheduled and triggered runs with retries and a run history.
- Problem it solves
- A long or repeated piece of work should run without someone keeping a laptop open.
- How it works
- A job is a set of tasks with a trigger and parameters. A wait for a person is modelled as a run that ends and a new run that starts on the decision.
- Small example
run one investigation per request, with two retries- In this agent
- Planned for running investigations.
- Usual confusion
- A job has no built-in way to wait for a human; design the wait.
5 · Watching and judging
MLflow tracing
Records every step of an agent run: each model call and tool call, with inputs, outputs and timings.
- Problem it solves
- A black-box agent cannot be debugged, audited or costed.
- How it works
- Decorate a function with @mlflow.trace, or turn on auto-logging for a model library, and each run appears as a tree of steps.
- Small example
one trace: model call, three tool calls, final answer, 31 seconds- In this agent
- The raw material for the eval harness, which answers 62 everyday ops questions about any agent from its traces.
- Usual confusion
- Tracing records what happened. Evaluation judges whether it was good.
MLflow evaluation and judges
Scores agent answers against expectations, with code checks or a model acting as judge.
- Problem it solves
- Without scores, you cannot tell whether a change made the agent better or worse.
- How it works
- You provide inputs, run the agent, and apply scorers: exact checks first, then judged qualities such as groundedness.
- Small example
does every number in the draft equal what the tools returned?- In this agent
- Deterministic checks first (the output contract and the grounding check, 48 unit tests), then a 20-case set of known signals and controls. Judged qualities come later, checked against human labels.
- Usual confusion
- A judge is a model and can be wrong. Calibrate it before trusting it.
System tables
Databricks' own usage, billing and audit records, as tables you can query.
- Problem it solves
- Cost and access questions need real records, not estimates.
- How it works
- Tables such as billing usage and audit events are maintained by the platform in a system catalog.
- Small example
SELECT usage_date, sum(usage_quantity) FROM system.billing.usage GROUP BY usage_date- In this agent
- Planned: cost per assessment from real billing rows.
- Usual confusion
- System tables are read-only records about the workspace.
Questions this page answers
Yes, in part. Mosaic AI is Databricks' name for its AI products as a group: Foundation Model APIs, Model Serving, Vector Search, Agent Framework, Agent Evaluation and AI Gateway. We use Foundation Model APIs today, and Vector Search, Agent Framework and evaluation are planned. Custom Model Serving is deliberately not used; AI Gateway is on for the model endpoint by default but we configure nothing on it. Databricks renames products often, so check the current docs.
Are we using Agent Bricks?No, on purpose. It builds common agents for you; we are building one by hand to learn the parts, and ours needs its own approval step and checks.
Why Unity Catalog functions and not letting the model write SQL?A function has typed inputs, a permission and an audit trail. A model that writes SQL can read anything its login can read, and can be talked into it by text in the data.
Why keep our own loop instead of Agent Framework's?The loop is where step limits, stop rules, approval and cost limits live, and we want them the same on every cloud. Agent Framework wraps the loop for packaging, tracing and registration.
Why not a live serving endpoint?Each investigation is reviewed by a person afterwards, so a job is enough. An endpoint adds cost and a way in.
What did the free workspace not allow?On one Free Edition workspace several calls were refused with 'try again later' on the day: Vector Search, serving and the SQL warehouse. A second Free Edition workspace allowed them, so the cause was a limit on the first workspace and not on Free Edition itself. A lesson in itself: test on more than one workspace before writing 'not supported'.
Where to go next
- The agent harness on Databricks: the 22 concerns bound to Databricks services, and one request end to end.
- The agent harness, end to end: the generic design this agent is built on.
- Claims FNOL assistant on Azure: the first agent on the same harness.
- The project page: tasks, roles and how to join.
- Databricks on this site: the platform map.