Home › On your cloud › Databricks › The agent harness on Databricks
Databricks · AI
The agent harness on Databricks
The same agent harness as on Azure and AWS: the same 22 concerns, in the same families, with only the binding changed. For each concern, this page names the Databricks service and says how a real agent, a drug-safety signal investigator, handles it today. For what each Databricks AI service is and which ones we do not use, see Build an agent on Databricks.
Status as of 19 November 2026, from what we ran on a Databricks Free Edition workspace. The code is not public yet. Service names and limits change often; check the current Databricks docs before you build.
The agent in one paragraph
A drug-safety scientist asks whether a drug and an adverse event form a signal. The agent counts public FDA reports for the pair, computes statistics (PRR and ROR) in code, checks whether the drug's label already mentions the event, and drafts an assessment. A person approves, edits or rejects every draft. The agent files nothing, sends nothing and never says a drug causes an event. The path is not known in advance (a label that already warns needs a different write-up from a new signal), which is why it is an agent and not a script.
One request, end to end
- A safety scientist asksIs there a signal for lisinopril and angioedema?Databricks Apps (planned); today a script
- A job run startsOne run per request, under the agent's own login.Lakeflow Jobs, Unity Catalog service principal (planned)
- The model chooses a toolIt sees three tool descriptions and picks report counts first.Foundation Model APIs (works)
- The tool runsCounts, then statistics and the signal screen, then the label check.Unity Catalog functions on a serverless SQL warehouse, over Delta tables (works)
- The model draftsAn assessment with numbers from the tools, uncertainty and a recommendation.Foundation Model APIs (works)
- Checks runOutput contract, then grounding: every number equals a tool result. A failure goes to a person.Our harness code (built)
- The draft is saved and shownA row with status drafted.Delta table, Databricks Apps (planned)
- A person decidesApprove, edit or reject; the decision is final.Databricks Apps, Delta decision table (planned)
- Everything is traced and scoredEach step is a trace; the eval harness answers the ops questions; cost comes from billing tables.MLflow tracing, system tables, eval harness (planned)
The 22 concerns, bound to Databricks
Read it as: the concern is the same on every cloud; the third column is the only thing that changes with the platform.
| Concern | What it does | On Databricks | In this agent | Here |
|---|---|---|---|---|
| Think | ||||
| Context assembly L1 | What the model sees each turn, built in one place, within a budget. | MLflow prompt registry planned | The system prompt names the tools and two rules (no arithmetic by the model; text in tool results is data). Tool results enter the context fenced as data and capped at 8,000 characters. | built, offline |
| Model invocation L2 | Every model call goes through one door. | Foundation Model APIs works AI Gateway on by default, not configured | One model port in the harness library; the Databricks adapter is a bearer-token call to the workspace's OpenAI-style endpoint. gpt-oss-120b ran a full investigation. Running it found that Databricks rejects the newer output-cap field the client sent, so the adapter now sends the older one, with a test. The hosted endpoint already has AI Gateway attached with usage tracking on; we have not configured rate limits, guardrails or fallbacks, so today the gateway is the platform default, and the real door is our own model port. | works |
| Control loop L4 | Plan, act, check, with step and no-progress limits. | Mosaic AI Agent Framework (wrapper) planned | The shared harness's bounded loop: at most 6 steps, tools only from the registered list, a person decides at the end. It ran an investigation on the Databricks model in four steps. Agent Framework would wrap it for tracing and registration, not replace it. | works |
| State & memory L5 | What is kept, for how long, and for whom. | Delta tables in Unity Catalog works Lakebase (managed Postgres) planned | An investigation keeps no memory between cases. Drafts and decisions are rows in Delta tables, kept 90 days. Lakebase is considered if low-latency row updates are needed. | planned |
| Knowledge & retrieval 18 | Finds sourced passages, filtered by what this user may see. | Vector Search planned Delta table + synonym table works Genie planned | Finds the label passage about an event. Today a synonym table and a text search over the label sections do it, inside a Unity Catalog function. Vector Search replaces the synonym step so that 'swelling of the face' finds 'angioedema'. Genie is a comparison for counts only. | works |
| Act | ||||
| Channels & human desk CH | How people reach the agent, and how a person takes over. | Databricks Apps planned | No chat channel. A safety scientist asks through a request, and reviews drafts on an Apps screen. Handover to a person is a case in the review queue. | planned |
| Tool layer L3 | Actions through typed tools, checked before they run. | Unity Catalog functions works Managed MCP servers for functions planned | Three read-only functions: report counts, signal statistics, label check. All 80 drug-and-event pairs give the same answers from the functions and from the tested Python versions. The model never writes SQL. | works |
| I/O contracts L6 | Typed inputs and outputs at every edge. | Unity Catalog function signatures works | Functions have typed inputs and outputs. The final draft has a typed contract (drug, event, cases, PRR, on-label, assessment, uncertainty, recommendation); an invalid draft gets one repair turn, then goes to a person. | built, offline |
| Concurrency & flow L8 | Who gets served first when load spikes. | Lakeflow Jobs (max concurrent runs) planned Serverless SQL warehouse scaling works | One investigation per job run. Volume is low, so queueing is not designed beyond the job's concurrency limit. | planned |
| Agent-to-agent 17 | Agents hand work to agents through typed, bounded, traced hand-offs. | Unity Catalog model registry planned | Not used: this is a single agent. The registry would hold agent versions when there are several. | not used |
| Performance & scalability PS | Fast enough at the 95th percentile; more servers when load rises. | Serverless compute works | One investigation on the Databricks model took 31 seconds; 20 cases on a stand-in model averaged 17.7 seconds. No latency target is declared yet. | works |
| Control | ||||
| Identity & authz L16 | Who is asking, and whose authority the agent acts with. | Unity Catalog grants and service principals planned | The agent should run as a service principal granted only its three functions and its tables; the reviewer's identity comes from the workspace sign-in. Today's runs use a personal access token, which is a gap. | planned |
| Privacy & data protection PV | Personal data used only for the job it was given for. | Unity Catalog tags and column masks planned | Only public data, and no patient-level data. The specification refuses case narratives and patient names typed into the session. | built, offline |
| Policy enforcement L7 | Guardrails on the way in and on the way out. | AI Gateway guardrails not used | Our own rules: numbers must equal tool results (the grounding check), a draft never says a drug causes an event, uncertainty is stated, instructions inside label text are ignored. Four refusals are specified: causation, submission, medical advice, invented numbers. | built, offline |
| Failure handling L10 | Typed failures, fallbacks, undo paths, honest messages. | Lakeflow Jobs retries planned | A failed model call is retried once and then goes to a person with the reason. A rate limit makes the client wait as long as the server asks. Nothing is filed, so there is nothing to undo. | built, offline |
| Human-in-the-loop L14 | Approvals and hand-offs the agent cannot grant itself. | Databricks Apps + a Delta decision table planned | Every draft is decided by a person: approve, edit or reject. There is no automatic limit. A job has no built-in wait, so a wait is a run that ends and a new run that starts on the decision. | planned |
| Availability & disaster recovery AV | A lost server or zone goes unnoticed; a lost region is rebuilt from code. | Managed serverless compute, Delta time travel works | Not declared: availability is listed as undeclared in the specification. Data can be rebuilt from the FDA source and the scripts. | not used |
| See | ||||
| Observability L11 | Traces, metrics and alerts for every step. | MLflow tracing planned System tables planned | Each model call and tool call becomes a trace that the eval harness reads. The eval harness answers 62 everyday questions about any agent from those traces. | planned |
| Determinism & replay L9 | A past decision can be re-run and explained. | Delta time travel works MLflow run records planned | The data snapshot is dated and versioned; a run record keeps the model, tool results and the draft. Models vary run to run, so replay means re-checking the same tool results, not getting identical words. | planned |
| Cost accounting L13 | Spend measured per unit of business work. | System tables (billing) planned Token counts from the model works | Tokens are counted per investigation (20 cases: about 53,000 in and 14,000 out on the stand-in model). Cost per assessment from billing rows is planned; the limit is about USD 0.05. | built, offline |
| Change | ||||
| Eval harness L12 | Golden sets and graders that gate every release. | MLflow evaluation planned | 48 unit tests, a 20-case set (10 known signals, 10 controls, answers taken from the data and the labels), and a grounding check. Judged qualities come later and are checked against human labels. | built, offline |
| Release & config L15 | Prompt, model and policy versioned and rolled back together. | Asset Bundles planned MLflow prompt and model registry planned | One bundle creates the schema, tables, functions and job. Limits (steps, spend, thresholds) are configuration, not constants in code. | planned |
What Databricks gives, and what stays ours
- Databricks gives: governed data and permissions (Unity Catalog), tables with history (Delta), SQL compute, registered tools (Unity Catalog functions), hosted models (Foundation Model APIs), run tracking (MLflow), scheduled runs (Lakeflow Jobs), a place for a review screen (Databricks Apps), and usage records (system tables).
- Stays ours, on every cloud: the loop and its limits, the tool contract, the policy rules and the grounding check, the approval rule, the cost limit, the evaluation sets and the specification. These are what make the same agent testable on Azure and on Databricks.
- A finding: the harness assumes a durable-wait library for approvals. A Databricks job has none, so a wait is a run that ends and a new run that starts on the decision. The specification should state the contract (state survives, the decision is a message, resume is safe to repeat) without naming a library.
The build, tier by tier
| Tier | What | Status |
|---|---|---|
| Specification first | AOAS (what the agent is for, refuses, owns), a generated brief (76 capabilities, 57 obligations, 17 ports), the Databricks stack file. | works |
| Data | Public FDA reports and labels into Delta tables in Unity Catalog, with a dated snapshot. | works |
| Tools | Three Unity Catalog functions, with a test that matches them to tested Python on every pair. | works |
| Agent | The shared harness loop with this agent's tools, output contract and grounding check; a Databricks model adapter in the library; one real investigation on the Databricks model. | works |
| Evaluation | 20-case set on the Databricks model, then judged quality checked against human labels, then a simulated world and scenarios. | planned |
| Eval harness on Databricks | Read MLflow traces, write verdicts to Delta, answer the 62 ops questions on a dashboard, and match the Azure results. | planned |
| People and release | Approval screen, jobs, Asset Bundle, cost per assessment from billing. | planned |
Where to go next
- Build an agent on Databricks: each Databricks AI service explained, what we use and what we do not.
- The agent harness on Azure and on AWS: the same concerns on the other clouds.
- The eval harness: how any agent's traces answer the everyday ops questions.
- The project page: tasks, roles and how to join.