Home › Learn › Databricks compute and cost › Warehouses and serverless

Databricks compute and cost · data engineering

SQL warehouses, serverless and who owns it: classic, pro, serverless, prod and dev, BU1 and BU2

Classic, pro and serverless SQL warehouses compared, serverless notebooks and jobs, performance versus standard mode, who owns serverless (account, workspace, prod and dev, business units), and which to pick.

Cluster anatomyWarehouses and serverlessChoosing computeCompute settingsCost by workloadLogs and visibilityPhoton and cachesHow billing worksDividing the billSystem tablesSharing and regionsBackup architectureCost-saving hacks

Prices, defaults, limits and preview status change often, and dollar figures here are illustrations. Check the current Databricks documentation and price list before quoting a number.

Warehouses and serverless

SQL warehouses in detail

A SQL warehouse is compute built only for SQL: BI dashboards, SQL editor, ad hoc queries, many users at once. It runs Spark underneath but hides it. You cannot run Python on it and you cannot tune most Spark settings. In return you get Photon on by default, a query queue, and concurrency scaling.

BI tool, SQLeditor, API Gateway + queue 1. checks result cache2. predicts query cost3. picks a cluster Result cachesame query, same data: instant SQL WAREHOUSE (size = T-shirt size, e.g. Small) Cluster 1 (always on) DriverPhoton Worker+ SSD cache Worker+ SSD cache Worker+ SSD cache more workers= bigger size Cluster 2 (added when queue grows) Identical copy. Handles more people at once,not a single faster query. Cluster 3 ... up to your "max clusters" Size up = one query runs faster. Max clusters up = more queries at once. run Workers read Delta/Parquet from cloud storage (below, not drawn)
Two dials, two different jobs. Cluster size (2X-Small up to 4X-Large, each step roughly double the machines) makes a single query faster. Number of clusters (min to max) lets more users run at the same time without queueing.

Classic vs Pro vs Serverless warehouse

FeatureClassicProServerless
Machines inYour accountYour accountDatabricks account
Photon engineYesYesYes
Disk (SSD) cacheYes, automaticYes, automaticYes, managed for you
Result cacheLocal to the warehouseLocal to the warehouseLocal plus a remote one that survives restarts (up to 24 hours)
Predictive IO (smarter file reads)NoYesYes
Intelligent workload management (predicts query cost, scales faster)No, simple queue-based scalingPartialYes, most aggressive
Start timeMinutesA few minutesSeconds
Best forCheap, steady, low-demand SQLHeavier SQL where you must stay in your networkAlmost everything, especially bursty BI
VerdictDrop for new work. Use only if serverless is blocked.Pursue only when serverless is not allowedPursue as the default

Predictive IO in plain words. Instead of reading a whole file and then filtering, the engine learns which parts of files a query will actually need and reads only those. It acts like an automatic index. It speeds up selective queries and MERGE/UPDATE/DELETE on Delta.

Intelligent workload management in plain words. Before a query runs, it is scored as cheap or heavy. Cheap queries slip through, heavy ones get capacity added early. This stops one huge query from blocking a dashboard.

Warehouses and serverless

Serverless that is not a SQL warehouse

Serverless notebooks, serverless jobs and serverless pipelines give you Spark without a cluster. You never pick machines, runtime or autoscaling. You write code, and Databricks supplies compute from a warm pool.

Classic notebook Notebookattached to cluster Your cluster: driver + workersyou chose size, runtime, configs, libraries Spark cache+ disk cache same JVM Serverless notebook / job Notebookthin client only Spark Connectremote protocol Databricks-managed Spark + Photonsize and scaling chosen automatically Warm VMpool plan only Because the client only sends a plan, the driver is not your notebook's process.That is why RDDs, some JVM tricks and df.cache() style features are restricted.
The big architectural change is Spark Connect. Your code builds a query plan and sends it over the network. Databricks owns everything on the other side, so it can resize, upgrade and swap machines under you.

Performance optimized vs standard mode

Serverless jobs and pipelines offer two modes. It is the same engine with a different promise about speed versus price.

Performance optimizedStandard
Start-upSeconds, VMs held readyMinutes, VMs allocated on demand
ScalingAggressive, scales fastMore conservative
Cost per hourHigherLower
Good forInteractive work, jobs with tight deadlines, short jobs where start-up dominatesNightly batch where nobody is waiting
VerdictPursue for notebooks and urgent jobsPursue for relaxed overnight jobs to save money

Serverless notebooks always behave as performance optimized because a person is waiting.

What you give up on serverless

What you gain. No capacity planning, instant start, nothing idle to forget, automatic upgrades, and the usual outcome that total cost goes down for bursty or short work.

Warehouses and serverless

Who owns serverless: accounts, workspaces, prod and dev, BU1 and BU2

There is no single "serverless switch" that creates one serverless thing. Serverless is a capability that an account turns on, and the compute itself is created per workspace, on demand, and isolated per customer. These are the layers, from the top.

CLIENT A: one Databricks ACCOUNT Billing, users, budgets, network connectivity configs (NCC), serverless terms are set here Unity Catalog METASTORE (one per region): catalogs bu1_prod, bu1_dev, bu2_prod, bu2_dev Workspace BU1-Prod Serverless SQL warehouse(s), tagged bu1/prod Serverless jobs and notebooksown isolated compute per run Workspace BU1-Dev Smaller warehouses, short auto-stop Serverless jobs and notebooksbudget policy with a lower cap Workspace BU2-Prod Serverless SQL warehouse(s), tagged bu2/prod Serverless jobs and notebooksown isolated compute per run Workspace BU2-Dev Smaller warehouses, short auto-stop Serverless jobs and notebooksbudget policy with a lower cap CLIENT BA different ACCOUNT Own workspacesOwn metastoreOwn serverless Never shares a VM, anetwork or a cache withClient A. Each customer'sserverless runs in itsown isolated sandbox ina Databricks-ownedcloud account.
Client A and Client B get completely separate serverless. Inside Client A, every workspace uses serverless independently, so prod and dev, or BU1 and BU2, are separated by workspace first and by warehouse, tag and budget policy second.
Your questionAnswer
At what level does the client turn serverless on?Two levels. The account admin accepts the serverless terms, creates network connectivity configs (NCC) and budget policies. Then each workspace admin allows serverless SQL warehouses, notebooks and jobs for that workspace. Serverless SQL warehouses are individual objects you create and size inside a workspace. Serverless notebooks and jobs have no object to create; compute appears when you run.
Will client A and client B get different serverless?Yes, always. They are different accounts, and Databricks isolates each customer's workloads from every other customer's. Nothing is shared at the VM level.
Can one client have two serverless, prod and dev?Yes. Use separate workspaces for prod and dev (strong boundary: access, networking, audit, catalog binding), and separate warehouses inside each. Each workspace's serverless is independent and has its own usage trail.
One for BU1 and one for BU2?Yes, in two ways. Hard split: a workspace per BU (or per BU and environment). Soft split: one workspace, one warehouse per BU, plus a serverless budget policy per BU that tags all of their serverless usage. Hard split gives clean isolation and simple bill reading. Soft split gives less sprawl but needs tag discipline.

Warehouses and serverless

Which one should I pick?

SituationPickWhyVerdict
Dashboards and BI over Delta tablesServerless SQL warehouseInstant start, result cache, scales with users, Photon + Predictive IOPursue
Same BI, but your network rules forbid serverlessPro SQL warehouseGets most features, stays inside your VPCPursue with a reason
Same BI on a classic warehouse by habitClassic SQL warehouseSlowest start, fewer featuresDrop
Nightly ETL, no one waitingServerless job (standard mode) or a job cluster with PhotonLowest cost for scheduled workPursue
ETL with a deadline or many short jobsServerless job (performance optimized)Start-up time dominates short runsPursue
Data science notebooks, explorationServerless notebook, or all-purpose cluster if you need custom libsFast to open, nothing to forget runningPursue
Need init scripts, custom JARs, Scala/RDD, special Spark configsClassic job or all-purpose clusterOnly classic gives that controlPursue
Heavy Python UDF pipelineClassic cluster, general or compute optimized, Photon off or on (little gain)Photon does not speed up UDFs. Rewrite as built-in SQL when possible.Pursue, and plan the rewrite
Model training on GPUsGPU ML runtime clusterOnly these have GPUsPursue
Team shares one big cluster 24 hours a dayAnything aboveIdle cost is the biggest hidden wasteDrop. Use auto-terminate or serverless.

FAQ

Questions people ask

Will client A and client B get different serverless?

Yes. They are different accounts and Databricks isolates each customer's serverless workloads from every other customer's. Nothing is shared at the VM level.

Can one client have separate serverless for prod and dev, or BU1 and BU2?

Yes. Use separate workspaces for hard separation, and separate warehouses plus serverless budget policies per business unit for soft separation. Each workspace uses serverless independently.

Bigger warehouse or more clusters?

A bigger size makes one query faster. More clusters let more queries run at once without queueing. They fix different problems.

Next

Where to go next

Related on this site: Which compute for which job · AQE, Photon, Predictive Optimization

All Databricks compute and cost pages · All data engineering · Databricks on this site