Home › On your cloud › Databricks › Build an agent

Databricks · AI

Build an agent on Databricks: the whole design, end to end

The purpose is two things at once: build a real agent on Databricks, and learn Databricks' AI offerings by using each one for a job. The agent is a drug-safety signal investigator: it counts adverse-event reports, computes statistics in code, checks the drug's label, and drafts an assessment for a person to decide. It follows the same seven parts as the claims (FNOL) guide.

Data in Delta: worksTools as Unity Catalog functions: worksDatabricks model runs the agent: worksRebuilt on the shared harness: worksTools as library tools call the Databricks functions: nextMLflow tracing: plannedVector Search: plannedApproval app: plannedDashboards: planned

Status as of 19 November 2026. The code is not public yet. Where this page says "works", we ran it on a Databricks Free Edition workspace.

What we use, what it is for, and how

ServiceWhat it is forHow this agent uses itStatus
Unity CatalogOne place that names, secures and tracks data, files, functions and models.Holds the schema, the tables, the raw-file volume and the agent's tools. The agent's login is granted only those tools and tables.works
Delta Lake tablesTables that are safe to update and keep their history.80 drug-and-event count rows and 10 drug labels, loaded from public FDA data.works
SQL warehouse (serverless)Compute that runs SQL and registered functions.Runs the table loads and the tool functions.works
Unity Catalog functionsSmall named functions, registered and permissioned, that an agent may call as tools.Three read-only tools: report counts, signal statistics, label check. The model can call these and nothing else, so it never writes SQL.works
Foundation Model APIsReady-hosted models you pay for per use, behind an OpenAI-style web address.The reasoning model (gpt-oss-120b) ran the agent loop with tool calls and wrote a draft assessment.works
MLflow tracing and evaluationRecords every step of a run, and scores answers.The recorder: each model call and tool call becomes a trace the eval harness reads. Also scores the 20-case test set.planned
Vector SearchFinds text by meaning, with an index that stays in step with a Delta table.Finding the label passage for an adverse event, so 'swelling of the face' can find 'angioedema'. A keyword-and-synonym lookup does the job meanwhile.planned
Mosaic AI Agent FrameworkDatabricks' standard way to package, register and deploy an agent so it can be traced and served.A wrapper around our own loop (the loop stays ours); gives it the standard interface, tracing and registration.planned
Databricks AppsSmall web apps that run inside the workspace, signed in as the user.The screen where the safety scientist approves, edits or rejects a draft.planned
Lakeflow JobsScheduled and triggered runs with retries.Runs an investigation as a job; a wait for a person ends the run and a new run starts on the decision.planned
System tablesDatabricks' own usage, billing and audit records as tables.Cost per assessment from real billing rows.planned
AI Gateway (default settings only)A control point in front of model endpoints: usage tracking, rate limits, guardrails, fallbacks.Usage tracking is on for the hosted model endpoint by default, so every call is counted by the platform. We configure nothing else on it.works
GenieAsk a governed table a question in plain English.A comparison, not part of the agent: what can Genie answer about the counts alone, and where does it stop?planned

What we are not using, and why

ServiceWhat it is forWhy we do not use it here
Agent BricksHigher-level builders for common agents (question answering over documents, information extraction, a supervisor over several agents), mostly without code.It hides the parts this build exists to learn, and our agent has a typed tool contract, an approval step and its own checks. Worth knowing for a customer who wants a quick answer over documents.
AI Functions (ai_query, ai_classify and similar)SQL functions that call a model once per row.Our agent decides which tools to call per case. It is not a per-row batch job. They would fit tagging thousands of report narratives.
Custom Model ServingHost your own model or agent as a web service.A person reviews each investigation later, so a job is enough and a live endpoint adds cost and a way in. Revisit when the agent is used interactively.
AI Gateway: limits, guardrails and fallbacksA control point in front of model endpoints.The hosted model endpoint we call already has AI Gateway attached with usage tracking on, by default. We have not configured rate limits, guardrails, request logging or fallbacks: our own harness enforces the step limit, the output contract and the grounding check. Worth configuring before more than one team uses the same endpoint.
Feature engineering, model training and fine-tuningBuild and tune your own models.There is no model to train: the statistics are formulas and the language model is used as it is.
Provisioned throughputReserved model capacity.Pay-per-use is enough at this volume.
Lakehouse MonitoringProfiles data and tracks drift in tables.Optional here; the eval harness watches the agent, not the tables.
Delta Sharing, Clean Rooms, MarketplaceShare data with other organisations.Nothing is shared.
Mosaic AI is Databricks' brand for the AI products as a group. So "are we using Mosaic AI?" is answered service by service in the two tables above. Product names change often; check the current Databricks docs before you build.

1 · Requirements

What it must do

How well

2 · Architecture

The idea

3 · Design

Security

Reliability

Cost

4 · Development

What exists

Honest note

5 · Testing

What ran

Still to do

6 · Deploy and release

Plan

7 · Monitor

Plan

Start here: the order to learn and build it

  1. Governed data. Create a schema, load a few tables, grant access. Draw the 2x2 table for one drug and one event by hand.
  2. Tools. Write one function that takes a drug and an event and returns the counts. Test it against a plain-Python answer.
  3. Model and loop. Call a hosted model with the function listed as a tool. Read one run's steps from start to finish.
  4. Checks. Add an output contract and a grounding check (every number equals a tool result). Break them on purpose with a scripted model.
  5. Tracing and evaluation. Trace a run in MLflow, then score 20 cases with known answers.
  6. People and release. Add the approval screen, run it as a job, and measure cost per investigation.

Each service explained

The same services, one at a time, in the order a request meets them. Names and limits change often; check the current Databricks docs before you build.

1 · Governed data

Unity Catalog

One place that names, secures and tracks everything: tables, files, functions and models.

Problem it solves
Without it, every tool has its own login and nobody can say who touched what.
How it works
Every object has a three-part name, catalog.schema.object. Permissions are granted on those names, and lineage records what read or wrote what.
Small example
GRANT SELECT ON TABLE workspace.drugsafety.pair_counts TO `agent-service-principal`
In this agent
Holds the schema, the tables, a volume for the raw files and the three tool functions. The agent's login is granted only those.
Usual confusion
It is not a database. It is the permission and lineage layer over data stored elsewhere.

Delta Lake tables

A table format that is safe to update and remembers its history.

Problem it solves
Plain files cannot be updated safely by two writers, and cannot answer 'what did this look like last week?'.
How it works
Data files plus a transaction log. Every change is a new version; older versions stay readable.
Small example
SELECT * FROM pair_counts VERSION AS OF 3
In this agent
Holds 80 drug-and-event count rows and 10 drug labels from public FDA data. Later: each draft assessment and each reviewer decision, so an audit can replay the past.
Usual confusion
A Delta table is not a different database. It is files in cloud storage with a log beside them.

SQL warehouse (serverless)

Compute that runs SQL and the registered functions.

Problem it solves
Something has to run the queries and functions; you do not want to manage a cluster to do it.
How it works
A serverless warehouse starts on demand and stops when idle. The Statement Execution API runs one SQL statement and returns rows.
Small example
POST /api/2.0/sql/statements with a warehouse id and a statement
In this agent
Runs the table loads, the function registration and every tool call.
Usual confusion
A warehouse is for SQL. Notebooks and jobs use other compute.

2 · Models

Foundation Model APIs

Ready-hosted models you call and pay for per use, with no servers to run.

Problem it solves
Hosting a large model yourself is slow and costly.
How it works
Each hosted model is an endpoint with an OpenAI-style chat format, including tool calls. Open models (gpt-oss, Llama, Qwen, Gemma) and embedding models (bge, gte) are listed in the workspace.
Small example
POST /serving-endpoints/databricks-gpt-oss-120b/invocations with a list of chat messages
In this agent
The reasoning model runs the loop: it chooses which tool to call, reads the results and drafts the assessment.
Usual confusion
Pay-per-token and provisioned throughput are two ways of buying the same models: share a pool, or reserve capacity.

Model Serving and AI Gateway

Host your own model or agent as a web service; AI Gateway puts limits, guardrails and logging in front of model endpoints.

Problem it solves
Without a control point, every team enforces its own rate limit, privacy rule and cost report.
How it works
Serving turns a registered model or agent into an endpoint that scales up and down. AI Gateway adds rate limits, guardrails, usage tracking and fallbacks in front of endpoints.
Small example
an endpoint with a rate limit of N calls per minute per user
In this agent
Custom Model Serving is not used: each investigation is reviewed by a person afterwards, so a job is enough. AI Gateway is attached to the hosted model endpoint with usage tracking on by default (we checked the endpoint); we have not configured rate limits, guardrails or fallbacks.
Usual confusion
Foundation Model APIs are hosted for you. Model Serving is where you host your own.

3 · Tools and knowledge

Unity Catalog functions as tools

A small named SQL or Python function, registered in Unity Catalog, that an agent may call.

Problem it solves
Letting a model write SQL is unsafe: it can read anything its login can, and text in the data can talk it into doing so.
How it works
A function has typed inputs, a comment the model reads as its description, a permission and an audit trail. The agent is shown only the functions registered for it.
Small example
CREATE FUNCTION workspace.drugsafety.report_counts(drug STRING, event STRING) RETURNS TABLE (...) RETURN SELECT ... WHERE t.drug = lower(report_counts.drug) AND t.event = upper(report_counts.event)
In this agent
The three tools: report counts, signal statistics (PRR, ROR and the screen, computed in SQL) and label check. A test runs all 80 drug-and-event pairs through the functions and through the tested Python versions: every answer matches.
Usual confusion
A tool is a contract. The model chooses a tool and its inputs, and the platform decides whether it may run.

Vector Search

Finds text by meaning, with an index that stays in step with a Delta table.

Problem it solves
Keyword search misses 'swelling of the face' when the label says 'angioedema'.
How it works
An embedding model turns text into a vector, a list of numbers where similar meanings sit close together. A query is embedded the same way and the nearest vectors win. A Delta Sync index re-embeds when the source table changes and inherits its permissions.
Small example
search the labels for 'swelling of the face' and get the angioedema passage
In this agent
Planned for finding the label passage about an event. Today a synonym table does the job, and the guide's next step swaps it.
Usual confusion
A vector index finds similar text, not true statements. A retrieved passage still has to be checked.

AI Functions

SQL functions that call a model once per row: ai_query, ai_classify, ai_extract and others.

Problem it solves
Tagging or extracting from thousands of rows should not need a loop in Python.
How it works
You call the function inside a SELECT, and the warehouse sends each row to a model endpoint.
Small example
SELECT ai_classify(narrative, ARRAY('serious', 'non-serious')) FROM reports
In this agent
Not used. The agent chooses its steps per case; it is not a per-row batch job. They would fit tagging thousands of report narratives.
Usual confusion
They are not an agent. They run the same model call on every row.

Genie

Ask a governed table a question in plain English and get SQL and an answer.

Problem it solves
Many people need an answer from a table and cannot write SQL.
How it works
A Genie space is configured with tables, instructions and example questions; it writes SQL, runs it and shows the result.
Small example
'How many reports of angioedema for lisinopril?' returns the count and the SQL behind it
In this agent
A comparison, not part of the agent: what Genie answers about the counts alone, and where it stops. It cannot weigh evidence or draft an assessment.
Usual confusion
Genie answers questions about data. It does not carry out a multi-step investigation with approval.

4 · The agent

Mosaic AI Agent Framework and Agent Bricks

Agent Framework is Databricks' way to write, log, test, register and deploy an agent. Agent Bricks builds common agents for you, mostly without code.

Problem it solves
Agents need a standard shape to be traced, evaluated and served consistently.
How it works
With Agent Framework you write the agent as a class with a standard interface (ResponsesAgent in current MLflow); MLflow logs it, Unity Catalog registers it, and a deploy step puts it behind an endpoint with a review app. Agent Bricks skips the code for document question-answering, information extraction and multi-agent supervision.
Small example
log the agent with MLflow, register it as workspace.drugsafety.signal_investigator, deploy it
In this agent
Agent Framework is planned as a wrapper around our own loop, for tracing and registration. Agent Bricks is not used: this build exists to see and control the parts.
Usual confusion
Agent Framework is not a loop engine like LangGraph. It standardises packaging, tracing and deployment.

Databricks Apps

Run a small web app (Streamlit, Dash, Gradio or FastAPI) inside the workspace, signed in as the user.

Problem it solves
You need a screen for people to act on what the agent produced, without hosting a server elsewhere.
How it works
You deploy app code; the platform runs it, handles sign-in and lets it query governed data on the user's behalf.
Small example
a page listing drafts, with Approve, Edit and Reject buttons that write a row
In this agent
Planned for the reviewer's screen: the decision is saved as a row in a Delta table.
Usual confusion
An App is a front end. The agent still runs as a job.

Lakeflow Jobs

Scheduled and triggered runs with retries and a run history.

Problem it solves
A long or repeated piece of work should run without someone keeping a laptop open.
How it works
A job is a set of tasks with a trigger and parameters. A wait for a person is modelled as a run that ends and a new run that starts on the decision.
Small example
run one investigation per request, with two retries
In this agent
Planned for running investigations.
Usual confusion
A job has no built-in way to wait for a human; design the wait.

5 · Watching and judging

MLflow tracing

Records every step of an agent run: each model call and tool call, with inputs, outputs and timings.

Problem it solves
A black-box agent cannot be debugged, audited or costed.
How it works
Decorate a function with @mlflow.trace, or turn on auto-logging for a model library, and each run appears as a tree of steps.
Small example
one trace: model call, three tool calls, final answer, 31 seconds
In this agent
The raw material for the eval harness, which answers 62 everyday ops questions about any agent from its traces.
Usual confusion
Tracing records what happened. Evaluation judges whether it was good.

MLflow evaluation and judges

Scores agent answers against expectations, with code checks or a model acting as judge.

Problem it solves
Without scores, you cannot tell whether a change made the agent better or worse.
How it works
You provide inputs, run the agent, and apply scorers: exact checks first, then judged qualities such as groundedness.
Small example
does every number in the draft equal what the tools returned?
In this agent
Deterministic checks first (the output contract and the grounding check, 48 unit tests), then a 20-case set of known signals and controls. Judged qualities come later, checked against human labels.
Usual confusion
A judge is a model and can be wrong. Calibrate it before trusting it.

System tables

Databricks' own usage, billing and audit records, as tables you can query.

Problem it solves
Cost and access questions need real records, not estimates.
How it works
Tables such as billing usage and audit events are maintained by the platform in a system catalog.
Small example
SELECT usage_date, sum(usage_quantity) FROM system.billing.usage GROUP BY usage_date
In this agent
Planned: cost per assessment from real billing rows.
Usual confusion
System tables are read-only records about the workspace.

Questions this page answers

Are we using Mosaic AI?

Yes, in part. Mosaic AI is Databricks' name for its AI products as a group: Foundation Model APIs, Model Serving, Vector Search, Agent Framework, Agent Evaluation and AI Gateway. We use Foundation Model APIs today, and Vector Search, Agent Framework and evaluation are planned. Custom Model Serving is deliberately not used; AI Gateway is on for the model endpoint by default but we configure nothing on it. Databricks renames products often, so check the current docs.

Are we using Agent Bricks?

No, on purpose. It builds common agents for you; we are building one by hand to learn the parts, and ours needs its own approval step and checks.

Why Unity Catalog functions and not letting the model write SQL?

A function has typed inputs, a permission and an audit trail. A model that writes SQL can read anything its login can read, and can be talked into it by text in the data.

Why keep our own loop instead of Agent Framework's?

The loop is where step limits, stop rules, approval and cost limits live, and we want them the same on every cloud. Agent Framework wraps the loop for packaging, tracing and registration.

Why not a live serving endpoint?

Each investigation is reviewed by a person afterwards, so a job is enough. An endpoint adds cost and a way in.

What did the free workspace not allow?

On one Free Edition workspace several calls were refused with 'try again later' on the day: Vector Search, serving and the SQL warehouse. A second Free Edition workspace allowed them, so the cause was a limit on the first workspace and not on Free Edition itself. A lesson in itself: test on more than one workspace before writing 'not supported'.

Where to go next