Home › Learn › Databricks compute and cost › Warehouses and serverless
Databricks compute and cost · data engineering
SQL warehouses, serverless and who owns it: classic, pro, serverless, prod and dev, BU1 and BU2
Classic, pro and serverless SQL warehouses compared, serverless notebooks and jobs, performance versus standard mode, who owns serverless (account, workspace, prod and dev, business units), and which to pick.
Prices, defaults, limits and preview status change often, and dollar figures here are illustrations. Check the current Databricks documentation and price list before quoting a number.
Warehouses and serverless
SQL warehouses in detail
A SQL warehouse is compute built only for SQL: BI dashboards, SQL editor, ad hoc queries, many users at once. It runs Spark underneath but hides it. You cannot run Python on it and you cannot tune most Spark settings. In return you get Photon on by default, a query queue, and concurrency scaling.
Classic vs Pro vs Serverless warehouse
| Feature | Classic | Pro | Serverless |
|---|---|---|---|
| Machines in | Your account | Your account | Databricks account |
| Photon engine | Yes | Yes | Yes |
| Disk (SSD) cache | Yes, automatic | Yes, automatic | Yes, managed for you |
| Result cache | Local to the warehouse | Local to the warehouse | Local plus a remote one that survives restarts (up to 24 hours) |
| Predictive IO (smarter file reads) | No | Yes | Yes |
| Intelligent workload management (predicts query cost, scales faster) | No, simple queue-based scaling | Partial | Yes, most aggressive |
| Start time | Minutes | A few minutes | Seconds |
| Best for | Cheap, steady, low-demand SQL | Heavier SQL where you must stay in your network | Almost everything, especially bursty BI |
| Verdict | Drop for new work. Use only if serverless is blocked. | Pursue only when serverless is not allowed | Pursue as the default |
Predictive IO in plain words. Instead of reading a whole file and then filtering, the engine learns which parts of files a query will actually need and reads only those. It acts like an automatic index. It speeds up selective queries and MERGE/UPDATE/DELETE on Delta.
Intelligent workload management in plain words. Before a query runs, it is scored as cheap or heavy. Cheap queries slip through, heavy ones get capacity added early. This stops one huge query from blocking a dashboard.
Warehouses and serverless
Serverless that is not a SQL warehouse
Serverless notebooks, serverless jobs and serverless pipelines give you Spark without a cluster. You never pick machines, runtime or autoscaling. You write code, and Databricks supplies compute from a warm pool.
Performance optimized vs standard mode
Serverless jobs and pipelines offer two modes. It is the same engine with a different promise about speed versus price.
| Performance optimized | Standard | |
|---|---|---|
| Start-up | Seconds, VMs held ready | Minutes, VMs allocated on demand |
| Scaling | Aggressive, scales fast | More conservative |
| Cost per hour | Higher | Lower |
| Good for | Interactive work, jobs with tight deadlines, short jobs where start-up dominates | Nightly batch where nobody is waiting |
| Verdict | Pursue for notebooks and urgent jobs | Pursue for relaxed overnight jobs to save money |
Serverless notebooks always behave as performance optimized because a person is waiting.
What you give up on serverless
- No choice of machine type, so no storage-optimized or memory-optimized picks.
- No init scripts or custom JARs on the machines. Libraries come through the environment settings.
- Most Spark settings are locked. Spark caching calls such as
cache()andpersist()are limited. - Unity Catalog is required, so legacy Hive metastore and DBFS mounts do not work.
- You cannot SSH or inspect a worker.
What you gain. No capacity planning, instant start, nothing idle to forget, automatic upgrades, and the usual outcome that total cost goes down for bursty or short work.
Warehouses and serverless
Who owns serverless: accounts, workspaces, prod and dev, BU1 and BU2
There is no single "serverless switch" that creates one serverless thing. Serverless is a capability that an account turns on, and the compute itself is created per workspace, on demand, and isolated per customer. These are the layers, from the top.
| Your question | Answer |
|---|---|
| At what level does the client turn serverless on? | Two levels. The account admin accepts the serverless terms, creates network connectivity configs (NCC) and budget policies. Then each workspace admin allows serverless SQL warehouses, notebooks and jobs for that workspace. Serverless SQL warehouses are individual objects you create and size inside a workspace. Serverless notebooks and jobs have no object to create; compute appears when you run. |
| Will client A and client B get different serverless? | Yes, always. They are different accounts, and Databricks isolates each customer's workloads from every other customer's. Nothing is shared at the VM level. |
| Can one client have two serverless, prod and dev? | Yes. Use separate workspaces for prod and dev (strong boundary: access, networking, audit, catalog binding), and separate warehouses inside each. Each workspace's serverless is independent and has its own usage trail. |
| One for BU1 and one for BU2? | Yes, in two ways. Hard split: a workspace per BU (or per BU and environment). Soft split: one workspace, one warehouse per BU, plus a serverless budget policy per BU that tags all of their serverless usage. Hard split gives clean isolation and simple bill reading. Soft split gives less sprawl but needs tag discipline. |
Warehouses and serverless
Which one should I pick?
| Situation | Pick | Why | Verdict |
|---|---|---|---|
| Dashboards and BI over Delta tables | Serverless SQL warehouse | Instant start, result cache, scales with users, Photon + Predictive IO | Pursue |
| Same BI, but your network rules forbid serverless | Pro SQL warehouse | Gets most features, stays inside your VPC | Pursue with a reason |
| Same BI on a classic warehouse by habit | Classic SQL warehouse | Slowest start, fewer features | Drop |
| Nightly ETL, no one waiting | Serverless job (standard mode) or a job cluster with Photon | Lowest cost for scheduled work | Pursue |
| ETL with a deadline or many short jobs | Serverless job (performance optimized) | Start-up time dominates short runs | Pursue |
| Data science notebooks, exploration | Serverless notebook, or all-purpose cluster if you need custom libs | Fast to open, nothing to forget running | Pursue |
| Need init scripts, custom JARs, Scala/RDD, special Spark configs | Classic job or all-purpose cluster | Only classic gives that control | Pursue |
| Heavy Python UDF pipeline | Classic cluster, general or compute optimized, Photon off or on (little gain) | Photon does not speed up UDFs. Rewrite as built-in SQL when possible. | Pursue, and plan the rewrite |
| Model training on GPUs | GPU ML runtime cluster | Only these have GPUs | Pursue |
| Team shares one big cluster 24 hours a day | Anything above | Idle cost is the biggest hidden waste | Drop. Use auto-terminate or serverless. |
FAQ
Questions people ask
Will client A and client B get different serverless?
Yes. They are different accounts and Databricks isolates each customer's serverless workloads from every other customer's. Nothing is shared at the VM level.
Can one client have separate serverless for prod and dev, or BU1 and BU2?
Yes. Use separate workspaces for hard separation, and separate warehouses plus serverless budget policies per business unit for soft separation. Each workspace uses serverless independently.
Bigger warehouse or more clusters?
A bigger size makes one query faster. More clusters let more queries run at once without queueing. They fix different problems.
Next
Where to go next
Related on this site: Which compute for which job · AQE, Photon, Predictive Optimization
All Databricks compute and cost pages · All data engineering · Databricks on this site