Home › Learn › Databricks compute and cost › Choosing compute
Databricks compute and cost · data engineering
Choosing Databricks compute: classic, serverless, SQL warehouses, users, batch and live access, settings and mistakes
How many kinds of serverless there are, what you can and cannot configure on each, the factors that decide the choice (users, concurrency, latency, batch versus live, language, networking), scenarios with verdicts, sizing, common mistakes and what to watch out for.
Prices, defaults, limits and preview status change often, and dollar figures here are illustrations. Check the current Databricks documentation and price list before quoting a number.
Start here
How many kinds of serverless are there?
"Serverless" is not one thing. Databricks uses the word for several separate products, and each has different settings. For your own code and SQL there are four. Other serverless features are billed and configured separately.
| Kind | What it runs | Who uses it |
|---|---|---|
| 1. Serverless SQL warehouse | SQL queries from the SQL editor, dashboards, Power BI and other BI tools | Analysts, BI tools, apps that send SQL |
| 2. Serverless notebooks | Python and SQL in an interactive notebook | Engineers and data scientists developing |
| 3. Serverless jobs | Scheduled or triggered tasks in Lakeflow Jobs | Production batch pipelines |
| 4. Serverless pipelines | Lakeflow Spark Declarative Pipelines | Declarative bronze, silver and gold |
| Other serverless features | Model Serving, Predictive Optimization, and others, each with its own endpoint or billing line | Specific features. Check each one's page and price. |
The first four share one idea: you do not pick machines, and Databricks runs the compute in its own account. Notebooks, jobs and pipelines all run on the same kind of serverless engine. A SQL warehouse is a separate product with its own sizing.
Enabling serverless for SQL warehouses is a separate step from enabling it for notebooks, jobs and pipelines, and availability depends on cloud and region. If a choice is greyed out in your workspace, ask your account admin before assuming it does not exist.
Configuration
What you can configure on each kind
| Kind | You can set | You cannot set |
|---|---|---|
| Serverless SQL warehouse | Cluster size (T-shirt size), the minimum and maximum number of clusters, auto-stop (5 minutes at the lowest in the UI, 1 minute through the API), tags, permissions, and a statement timeout at session or workspace level | Machine type, Photon on or off (it is always on), spot, init scripts, which VPC it runs in |
| Serverless notebook | Environment version, base environment and libraries, and memory (standard is 16 GB, high is 32 GB, with a higher DBU rate for high memory) | Machine type, number of workers, most Spark settings, compute policies |
| Serverless job | Performance mode (performance optimized or standard), environment and libraries, job timeout, retries, notifications, tags and a budget policy | Machine type, number of workers, autoscaling limits, Photon on or off, init scripts, custom images |
| Serverless pipeline | Serverless on or off, triggered or continuous mode, performance mode, and a budget policy | Machines and worker counts |
The pattern is simple: on serverless you set behaviour and limits (mode, timeout, memory, libraries, tags), and you give up machine-level control (node type, worker count, spot, init scripts, Photon). Only warehouses give you a sizing knob, because a warehouse is sized by T-shirt size and cluster count.
Two settings are worth knowing exactly: serverless jobs in Standard mode use fewer DBUs but typically take 4 to 6 minutes to start, and auto-stop on pro and classic warehouses defaults to 45 minutes with a 10-minute minimum, while serverless defaults to 10 minutes. The page on cost-saving hacks lists the rest.
Factors
The factors that decide the choice
| Factor | Ask | Leans toward |
|---|---|---|
| Workload shape | Scheduled batch, interactive, live BI queries, or always-on streaming? | Batch: jobs compute or serverless jobs. Interactive: serverless notebook. BI: SQL warehouse. Always-on streaming: classic job cluster. |
| Concurrency | How many queries run at the same moment, not how many people exist? | Many concurrent queries: a warehouse with more clusters |
| Latency | Does a user wait for the first result? Is a start-up of minutes acceptable? | Seconds: serverless. Minutes fine: classic, or serverless Standard mode. |
| Predictability | Steady 24 hours a day, or bursty? | Bursty: serverless, because idle time is cheap. Steady and heavy: classic on spot or committed VMs can win. |
| Language and API | Scala, R, RDD code, continuous streaming triggers? | Classic or dedicated compute. Serverless restricts these. |
| Libraries and customisation | Custom images, init scripts, special drivers? | Classic |
| Networking | Must the compute sit inside your private network, for example to reach a database that only has a private IP, or a firewall that allows only your own IP ranges? | Serverless runs in Databricks' network, not yours. It can reach private resources only through a network connectivity configuration (private endpoints, stable IP ranges, outbound policies), which has limits: for example, firewall enablement is not supported for S3 or DynamoDB. If you cannot or may not set that up, use a pro warehouse or classic compute. |
| Run length | Could a run exceed 7 days? | Classic. Serverless stops runs after 7 days. |
| Cost attribution | Do you need to trace each VM on the cloud bill? | Classic, where tags reach the VMs. Serverless uses budget policies. |
| Team skill | Is anyone available to tune clusters? | No one: serverless |
| Availability | Is serverless enabled in your cloud and region? | If not, pro warehouses and classic compute |
Scenarios
Which compute for which job
| Scenario | Pick | Why | Verdict |
|---|---|---|---|
| Scheduled ETL, nothing unusual | Serverless jobs, Standard mode if it can wait | No cluster to tune, and idle time is not billed | Pursue |
| Large steady ETL, Scala or RDD code, spot savings wanted | Classic job cluster | Full control and the lowest hourly rate on spot | Pursue |
| A few analysts running ad-hoc SQL | One small serverless SQL warehouse, short auto-stop | Starts in seconds and costs almost nothing idle | Pursue |
| Dashboards for many users, Power BI | Serverless SQL warehouse with a higher maximum cluster count, one per business unit | Concurrency scales out, and cost is attributable | Pursue |
| Live, always-on application queries | A warehouse sized for the peak, or a database designed for point lookups | Warehouses are for analytic queries. Many tiny key lookups are a different workload. | Pursue with a load test |
| Notebook development | Serverless notebook, or standard all-purpose compute | Fast start for one user | Pursue |
| GPU, RDD, R, or privileged machine access | Dedicated all-purpose compute | These need dedicated access mode | Pursue |
| Always-on low-latency streaming | Classic job cluster, continuous | Serverless does not support the continuous and processing-time triggers | Pursue |
| Custom networking or hybrid connections | Pro warehouse or classic compute | These run in your own cloud network | Pursue |
| Declarative pipelines | Serverless pipelines, triggered mode | Managed scaling, no always-on cluster | Pursue |
| Scheduled production job on an all-purpose cluster | Move it to jobs compute or serverless jobs | All-purpose carries a higher DBU rate | Drop |
| One huge warehouse for every team | Separate warehouses per business unit | No attribution, and teams slow each other down | Drop |
More detail on the warehouse types: SQL warehouses and serverless. Interview-style version: which compute for which job.
Sizing
Number of users, concurrency and performance
Count concurrent queries, not users. Ten users can mean two queries or two hundred. A dashboard with twelve tiles sends twelve queries each time it opens or refreshes.
| Problem you see | What it means | Fix |
|---|---|---|
| One query is slow, or spills to disk | The cluster is too small for that query | Increase the size |
| Many queries wait in a queue | Too few clusters for the concurrency | Raise the maximum number of clusters |
| Cost is too high, queries are fast | The warehouse is bigger than it needs to be | Step down one size, as long as nothing spills or slows noticeably |
- On classic and pro warehouses, the planning rule is about 10 concurrent queries per cluster. Autoscaling adds clusters based on the estimated time to clear running and queued queries, and a query queued for 5 minutes triggers scale-up.
- Serverless warehouses scale on the time needed to process running, queued and expected incoming queries (looking about two minutes ahead), and scale down after 15 minutes of low load while keeping enough clusters for the last 15 minutes' peak.
- Queries queue when every cluster is full or the warehouse is starting. A queue holds up to 1,000 queries.
- Watch the Peak Queued Queries metric on the warehouse monitoring page. A value that is consistently above zero means more size or more clusters.
- Databricks suggests starting a serverless warehouse larger than you think you need and sizing down as you test.
- Pro and classic warehouses take several minutes to start (typically about 4). Serverless starts in seconds. Serverless also has Intelligent Workload Management, which pro and classic lack, and pro adds Predictive IO but classic has Photon only.
Treat the 10-per-cluster figure as a planning rule, not a hard limit, and load test with your real dashboards.
Mistakes
Common mistakes
| Mistake | Why it hurts | Fix |
|---|---|---|
| Scheduled jobs left on all-purpose clusters | Higher DBU rate, and the cluster often stays up | Jobs compute or serverless jobs |
| Making the warehouse bigger to fix a queue | Size helps one query. Queues need more clusters. | Raise the maximum cluster count |
| Dashboards refreshing every few minutes | Each refresh keeps the warehouse awake all day | Refresh less often, and use result caching and precomputed tables |
| Choosing serverless for work it cannot run | No RDD APIs and no sparkContext, no continuous streaming triggers, a 7-day limit | Check the limits first, or use classic |
| One warehouse shared by every team | No cost attribution, and noisy neighbours | One warehouse per business unit |
| Ignoring the queue metric | Users blame the tool while queries wait | Alert on Peak Queued Queries |
| Long auto-stop on pro or classic warehouses | The default is 45 minutes of idle billing | Shorten it, or use serverless |
| Picking a pro warehouse when serverless is available | Slow start and no Intelligent Workload Management | Use pro only for custom networking or when serverless is unavailable |
| Dedicated access mode by default | Databricks recommends standard for most work. Dedicated is for RDD, GPUs, R and privileged access. | Start with standard |
| Turning Photon on or off without measuring | It raises the DBU rate, and helps only some workloads | Test your real job |
| Assuming serverless is always cheaper | Its DBU rate is higher. It saves idle time and VM work, not every dollar. | Compare total monthly cost for your usage pattern |
| Treating serverless as zero-effort | A newer environment version can change behaviour, for example how errors surface under Spark Connect | Pin the environment version and test before upgrading |
| No tags or budget policy on serverless | Usage cannot be split by team afterwards | Set a budget policy on every serverless workload |
Watch out
What to watch out for
- Start-up time. Serverless in default mode starts quickly. Standard mode, pro and classic take minutes. Do not put a slow start in front of a user.
- Serverless limits. Only Spark Connect APIs, no RDD access, no global temporary views, 7-day maximum runs. The full list is on the limitations section of the comparison page.
- Quotas. Serverless usage is limited in DBUs per hour, so a big backfill can hit a quota.
- Opaque cost. You see DBUs, not machines. Use system tables and budget policies for attribution.
- Availability. Check that serverless is enabled for your cloud, region and each product kind.
- Environment versions. They change what libraries and behaviour you get. Pin and test.
- Dev and prod. Keep separate warehouses and workspaces so a dev experiment cannot slow production dashboards.
Quick chooser
Six questions, in order
- Is it SQL from a BI tool or editor? Use a serverless SQL warehouse. Use pro or classic only for custom networking or when serverless is unavailable.
- Is it interactive development? Use a serverless notebook, or standard all-purpose compute. Use dedicated for GPU, RDD or R.
- Is it a scheduled job? Use serverless jobs unless a limit blocks it. Use Standard mode if it can wait.
- Does it need Scala, RDD code, custom images, special networking, always-on streaming, or runs over 7 days? Use a classic job cluster.
- Is it large, steady and heavy? Price a classic cluster on spot against serverless for your real usage.
- Is it a declarative pipeline? Use serverless pipelines in triggered mode.
Worked decision
A shared dev setup for data engineers and data scientists, available any time
"Anytime they can query" does not mean "a cluster running 24 hours a day". It means a user can start work whenever they want and not wait long. Compute that starts in seconds, or restarts on demand, gives the same availability without paying for the empty hours.
| Option | How it works | Illustrative monthly cost | Verdict |
|---|---|---|---|
| A. One big shared cluster running 24 hours a day | 8 machines (16 DBU per hour) on all-purpose compute, always on | 730 h × 16 = 11,680 DBU × $0.55 = $6,424, plus VMs $2,243 = $8,667 | Drop. Most of those hours are empty. |
| B. One shared cluster that starts on demand and stops when idle | Standard access mode, autoscaling from 2 to 8 machines, auto-terminate after 30 to 60 minutes. Users reconnect and it restarts. | Active 12 h a day for 22 days at an average of 4 machines: 264 h × 8 DBU × $0.55 = $1,162, plus VMs $406 = $1,567 | Pursue when many users run light work and need Scala, RDD or libraries |
| C. Serverless notebooks, one per person | No cluster to wait for or share. Each person's session is isolated. | You pay only while a session is active. At an assumed 2 DBU per active hour, 88 hours a month is 176 DBU × $0.75 = $132 per person | Pursue as the default for DE and DS |
| D. One small serverless SQL warehouse per team | 2X-Small (4 DBU per hour), starts in seconds, short auto-stop. Anyone can run SQL at any time. | 6 busy hours a day for 22 days: 132 h × 4 = 528 DBU × $0.70 = $370 | Pursue for ad-hoc SQL |
| E. Personal single-node classic clusters | One small machine per person, policy-limited, auto-terminate after 20 to 30 minutes | 88 active hours × 2 DBU × $0.55 = $97, plus VMs $34 = $131 per person, before idle tails | Pursue for work serverless cannot do |
| F. An instance pool to speed up classic starts | Idle VMs wait in a pool so a cluster starts faster | No DBUs for idle pool machines, but your cloud bills them while they sit | Pursue only with a small minimum idle count |
The cheapest effective setup
- Serverless notebooks for most DE and DS work (option C), because nothing runs, or bills, when nobody is working.
- One small serverless SQL warehouse per team (option D) for anyone who just wants to query.
- Personal single-node classic clusters or a shared standard cluster (E or B) only for what serverless cannot do: Scala or RDD code, custom images, GPUs, or private-network access.
- Run heavy or scheduled work as a job, not on the dev cluster. See the nightly ETL decision on the cost map.
Guardrails that matter more than the choice
- A cluster policy that forces auto-termination (60 minutes at most), caps the number of machines, allows only a few machine types, and requires tags.
- A dev catalog with sampled or synthetic data, so nobody scans production-sized tables to try an idea.
- A budget policy and tags on every serverless workload, so dev spend is visible by team.
- An alert on anything running longer than a set number of hours, using the compute system tables.
- Never put a scheduled production job or a refreshing dashboard on a dev cluster. Activity keeps an idle cluster from auto-terminating.
What goes wrong
| Problem | Why | Fix |
|---|---|---|
| The "shared" cluster never stops | A forgotten loop, streaming query or auto-refresh counts as activity, so auto-termination never triggers | Policy, alerts, and no scheduled work on dev compute |
| One user's heavy query slows everyone | A shared cluster has shared memory and cores | Serverless or personal compute for heavy work, and a sampled dev catalog |
| Library conflicts on a shared cluster | Everyone installs into one environment | Notebook-scoped libraries, or serverless environments per person |
| Standard mode blocks something people need | Standard access mode has no R, and limits on RDD and Spark context | Dedicated mode for those users only |
| Classic start-up kills the "anytime" promise | A cold classic cluster takes minutes | Serverless, or a small pool |
| Serverless cannot reach a private dev database | It runs in Databricks' network | A network connectivity configuration, or classic compute for that user |
The figures use the same illustrative assumptions as the cost map: 2 DBU per machine per hour, $0.384 per machine per hour in cloud VMs, and third-party summaries of list rates. Serverless DBU use per session is an assumption.
Checked
What was checked, and what was not
These were checked against Databricks documentation on 21 November 2026:
- Serverless compute overview (notebooks, jobs, pipelines, SQL warehouses, and separate enablement for warehouses)
- SQL warehouse types (serverless, pro and classic features and start-up)
- Warehouse sizing, scaling and queuing (10 queries per cluster, queue limit, scale-up and scale-down rules)
- Serverless environment, dependencies and memory
- Access modes and limitations (standard and dedicated)
- Serverless limitations and performance modes
Not confirmed: whether high memory is still labelled preview, the exact minimum and maximum cluster controls shown on a serverless warehouse in your workspace, the default auto-termination time of all-purpose clusters, which release channels apply to serverless warehouses and pipelines, and the full list of other serverless features. Check the settings page in your own workspace before you rely on any of them.
FAQ
Questions people ask
How many types of serverless does Databricks have?
Four for your own code and SQL: serverless SQL warehouses, serverless notebooks, serverless jobs and serverless pipelines. Other features such as Model Serving and Predictive Optimization are separate serverless products with their own settings and billing.
What can I configure on serverless jobs?
Performance mode, the environment version and libraries, the job timeout and retries, notifications, tags and a budget policy. You cannot choose machine types, worker counts, Photon or init scripts.
A warehouse is slow. Do I make it bigger or add clusters?
Make it bigger when a single query is slow or spills to disk. Add clusters when queries wait in a queue. Size helps one query, clusters help concurrency.
When should I use classic compute instead of serverless?
When you need Scala or RDD code, custom images or init scripts, custom networking, always-on low-latency streaming, runs longer than 7 days, or the lowest hourly price on spot for large steady work.
Next
Where to go next
Related on this site: Which compute for which job · Biggest cost and performance levers · Tagging best practices · Cost and FinOps for the platform
All Databricks compute and cost pages · All data engineering · Databricks on this site