Home › Learn › Databricks compute and cost › Compute settings
Databricks compute and cost · data engineering
Databricks compute settings reference: SQL warehouse types, job clusters, node types, access modes and wrong settings
Every kind of compute in one map, then the settings for SQL warehouses (serverless, pro, classic), serverless and classic jobs, all-purpose clusters and node types, with the good value, the wrong value and what to check.
Prices, defaults, limits and preview status change often, and dollar figures here are illustrations. Check the current Databricks documentation and price list before quoting a number.
The map
Every kind of compute, in one table
Three words that are not the same thing. Classic vs serverless says who runs the machines: you (in your cloud account) or Databricks (in its account). All-purpose vs jobs vs SQL warehouse says what the compute is for. Interactive is just a way of working, typing in a notebook, and is not a product. So an all-purpose cluster is a classic cluster used for interactive work. A job cluster is also classic, but it is not all-purpose. A serverless notebook is interactive, but it is not classic.
There are three families: SQL warehouses (for SQL and BI), jobs compute (for scheduled work) and all-purpose compute (for interactive work). Each exists in a classic form that runs in your own cloud account, and most have a serverless form that runs in Databricks' account.
| Compute | Classic form | Serverless form | Used for |
|---|---|---|---|
| SQL warehouse | Classic and Pro warehouses (VMs in your account) | Serverless warehouse | SQL editor, dashboards, Power BI, apps sending SQL |
| Jobs compute | Job cluster, created for one run and deleted after | Serverless jobs | Scheduled and triggered batch work |
| All-purpose compute | All-purpose cluster, in standard or dedicated access mode | Serverless notebooks | Interactive development, exploration, ML |
| Pipelines | Classic pipeline compute | Serverless pipelines | Declarative bronze, silver and gold |
Yes, classic warehouses still exist. I found no deprecation notice. The three warehouse types are serverless, pro and classic. For choosing between them, see choosing compute. This page is the settings reference: what each setting does, what to set and what people set wrongly.
Warehouses
SQL warehouse settings: serverless, pro and classic
Which type, and why classic still exists
| Serverless | Pro | Classic | |
|---|---|---|---|
| Runs in | Databricks' account | Your cloud account | Your cloud account |
| Start-up | Seconds | Several minutes (typically about 4) | Several minutes (typically about 4) |
| Photon | Yes | Yes | Yes |
| Predictive IO | Yes | Yes | No |
| Intelligent Workload Management | Yes | No | No |
| Scaling | Automatic, within your minimum and maximum | Slower to scale up and down | Slower to scale up and down |
| Why choose it | The default choice when it is available | Custom networking, federation or hybrid connections, or serverless not available | Basic interactive exploration when neither other type is possible |
- In the UI, serverless is the default where your region supports it.
- Through the API, the default type has been classic. A warehouse created from Terraform or a script without the type set can silently be a classic one. To get serverless, set the type and the serverless flag explicitly (the docs describe
warehouse_typeandenable_serverless_compute). - Classic and pro warehouses need enough vCPU quota in your cloud account. If quota is short, the warehouse fails to start.
The settings
| Setting | What it does | Good value | Wrong value |
|---|---|---|---|
| Cluster size | The size of each cluster, from 2X-Small (one worker) to 4X-Large (256 workers). It sets how fast one query runs. | Start small and raise it only if a query is slow or spills to disk. Databricks suggests starting a serverless warehouse larger and stepping down while testing. | Leaving a large default for light queries, or raising size to fix a queue |
| Scaling: minimum and maximum clusters | How many clusters run at once. This sets concurrency. The default is one and one. | Maximum sized to the real peak. Rule of thumb: one cluster per 10 concurrent queries. | Maximum of 1 with many dashboard users, or a very high maximum with no cost alert |
| Auto stop | Minutes of idle time before it stops | Serverless: the default is 10 minutes, the UI minimum is 5, the API allows 1. Pro and classic: default 45, minimum 10. Shorten for ad-hoc use. | Leaving 45 minutes on a pro or classic warehouse |
| Warehouse type | Serverless, pro or classic | Serverless unless you need networking control | Pro by habit, or classic by an API default |
| Statement timeout | Stops a query that runs too long. The default is 2 days. Set at session or workspace level, with a warehouse-level setting in Beta. | A limit that suits your longest legitimate query | Leaving the 2-day default |
| Permissions | Who can use or manage the warehouse | Use for the team that owns it, manage for few | Everyone can manage |
| Tags | Attach cost attribution labels | Team, environment and owner on every warehouse | No tags, so the bill cannot be split |
| Default warehouse | An admin can set a workspace-wide default warehouse for ad-hoc queries (introduced in 2026). It works with all three types. | Point it at a small serverless warehouse | Leaving ad-hoc users on a large shared one |
Not confirmed from the docs I could search: the Advanced options in the create form, which include the spot instance policy (pro and classic only), the release channel (current or preview) and tags. Open the form in your workspace and read each option's help text before relying on my description.
Classic vs pro
Classic SQL warehouse vs pro SQL warehouse, in detail
Both run on machines in your cloud account, both take several minutes to start (typically about 4), and both scale less quickly than serverless. The documented differences are small, and the price per DBU is not.
| Classic | Pro | |
|---|---|---|
| Photon engine | Yes | Yes |
| Predictive IO (faster reads, fewer files scanned) | No | Yes |
| Intelligent Workload Management | No | No |
| Materialized views, streaming tables and query federation | The official pricing page lists these under Pro and Serverless. A third-party comparison says classic lacks them. I could not confirm classic's status, so check the docs. | Yes |
| Illustrative rate per DBU | about $0.22 | about $0.55 |
| Intended use, per Databricks | Basic interactive exploration when neither serverless nor pro is possible | When serverless is not available, or you need custom networking (for example to connect other services in your network, or hybrid and on-premises connections) |
The settings are the same for both
Cluster size, minimum and maximum clusters, auto stop (default 45 minutes, minimum 10), warehouse type, tags, permissions, statement timeout, and the Advanced options. Spot instance policy applies only to the types that run on your machines, so it is the one setting that serverless does not have. I could not confirm its exact options from the docs I could search.
What decides it
| Factor | Leans toward |
|---|---|
| You only run basic interactive queries and want the lowest DBU rate | Classic |
| Large scans where faster reads matter (Predictive IO) | Pro |
| You need materialized views, streaming tables or query federation in the warehouse | Pro (confirm classic first) |
| You need the warehouse inside your private network | Pro |
| Serverless is available and your networking allows it | Neither. Use serverless. |
| Burst use, many short sessions | Neither. Start-up of about 4 minutes and a 45-minute default stop make both costly. |
Cost example
A Medium warehouse burns 24 DBU per hour. At the illustrative rates, the DBU part is $5.28 per hour on classic ($0.22 × 24), $13.20 on pro ($0.55 × 24) and $16.80 on serverless ($0.70 × 24). Classic and pro also bill the same cloud machines, so the VM part is added to both and the gap between them is only the DBU difference: $7.92 per hour, or about $697 over 88 busy hours. Serverless includes the machines and adds no idle tail beyond its short auto-stop. Pro is worth that extra DBU cost only when it gives you a feature you need.
Not confirmed: whether Python UDFs, materialized views and streaming tables work on classic warehouses today, and the exact spot policy options. The rates are illustrative third-party summaries. Check the warehouse types page and your price list.
Jobs
Jobs: serverless or classic job cluster
| Use | When |
|---|---|
| Serverless jobs | Python or SQL tasks, no special needs, bursty or modest volume, you do not want to tune a cluster. Use Standard performance mode when a start-up of 4 to 6 minutes is fine. |
| Classic job cluster | Scala or RDD code, custom images or init scripts, custom networking, always-on streaming, runs longer than 7 days, specific machine types, or large steady work where spot pricing wins. |
Serverless job settings
Performance mode, environment version and libraries, job timeout, retries, notifications, tags and a budget policy. There are no machine settings. See choosing compute.
Classic job cluster settings
| Setting | What it does | Good value | Wrong value |
|---|---|---|---|
| Databricks Runtime version | Spark, Python and library versions | A current long-term-support version, pinned | An old runtime that is out of support, or "latest" that changes under you |
| Photon | A faster engine at a higher DBU rate | On for SQL-heavy and large scans, after a test | On for Python UDF code, or never tested |
| Worker node type | The machine family and size for workers (see the node types section) | Match the bottleneck: memory for spill, compute for CPU | The largest type "to be safe" |
| Driver node type | The machine for the driver. It can differ from workers. | Smaller than workers unless you collect big results | A big driver on every job, or a tiny one with collect() |
| Workers: fixed or autoscaling | A fixed count, or a minimum and maximum | Autoscaling for batch with uneven stages. Fixed size for continuous streaming. | Autoscaling on a streaming job (Databricks advises against it, because it is slow to shrink), or autoscaling on a spark-submit job, which does not support it |
| Availability (spot) | On-demand, spot, or spot with fallback. first_on_demand sets how many of the first nodes, starting with the driver, are on-demand. | Spot with fallback, first_on_demand of 1 to keep the driver on-demand | All spot for a job that cannot restart, or all on-demand for large fault-tolerant batch |
| Access mode | Standard (shared) or dedicated (single user or group) | Standard for most work | Dedicated by default |
| Spark configuration | Overrides Spark defaults | Few, documented and justified | Copied lists from old blog posts that fight AQE |
| Tags and policy | Cost labels, and guardrails on what users can create | Required tags and a policy that caps size | No policy, so anyone can launch a huge cluster |
| Timeout and retries | Stop a hung job, and rerun a failed one | A timeout and a small retry count (the timeout applies to each retry) | No timeout, or max_retries of -1 |
| Reusing a job cluster across tasks | Pays start-up once for several tasks | Share it for similar tasks | A separate cluster for every small task |
Server types
Node types: which machine for which problem
On AWS, Databricks node types follow the EC2 families. Other clouds have equivalents with different names. Databricks also offers fleet instance types, which can draw from several similar machines to reduce capacity errors.
| Family | Use it when | Notes |
|---|---|---|
General purpose (m-family, fleet: md-fleet) | You are unsure, or the job is balanced | A safe starting point |
Memory optimized (r-family, fleet: rd-fleet, rgd-fleet) | The Spark UI shows spill to disk, wide joins, or big aggregations | Often better than adding more general-purpose nodes |
| Compute optimized (c-family) | The job is CPU-bound and does little shuffling | Too little memory for large joins |
| Storage optimized (i-family) | Read-heavy and interactive work that benefits from local SSD and the disk cache | I did not find a Databricks fleet type for this family |
| GPU (g and p families) | Model training and some inference | Not available as fleet types. Pick a specific GPU type, usually on dedicated access mode. |
| Graviton (Arm) | Plain batch jobs, to cut the VM price | Documented as unsupported for Databricks SQL, declarative pipelines and Container Services. Python UDFs need runtime 15.2 or later. Not offered as a fleet type. |
- A
din the name (for examplemd-fleet) means local SSD, which the disk cache uses. - When the worker type is a fleet type, the spot bid percentage has no effect.
- Single node compute runs everything on the driver. Use it for small exploration, not for jobs that need to scale.
| Symptom in the Spark UI | Try |
|---|---|
| Spill to disk, out-of-memory errors | Memory optimized nodes, or fewer cores per node |
| CPU at 100 percent, little shuffle | Compute optimized nodes |
| Repeated scans of the same data | Nodes with local SSD, so the disk cache helps |
| Nodes below about 30 percent CPU and memory | Fewer or smaller nodes |
| One slow task while others finish | Skew. A different node type will not fix it. |
All-purpose
All-purpose compute: access modes and the usual mistakes
- Standard access mode (formerly called shared) supports Python, SQL and Scala on recent runtimes, and Databricks recommends it for most workloads. It has limits: no R, no Hive UDFs, no
scorsqlContextin Scala. - Dedicated access mode (formerly single user) is for RDD APIs, GPUs, R and privileged machine access.
- Auto-termination: a value of 0 in a policy means never. I could not confirm the default for all-purpose compute, so check it in your workspace.
| Mistake | Why it hurts | Fix |
|---|---|---|
| Auto-termination off or set to many hours | An idle cluster bills all night and at weekends | A policy that forces a short auto-termination |
| A big cluster for a notebook that reads a small table | Pays for unused cores | Single node, or serverless notebooks |
| Scheduled jobs left on an all-purpose cluster | Higher DBU rate | Move to jobs compute or serverless jobs |
| Dedicated mode by default | Fewer users per cluster and more clusters | Standard unless you need RDD, GPU or R |
| No cluster policy | Anyone can launch an expensive cluster | Policies that cap size, force tags and set auto-termination |
Find the wrong settings in your own workspace
SELECT cluster_id, cluster_name, auto_termination_minutes, worker_node_type, min_autoscale_workers, max_autoscale_workers FROM system.compute.clusters WHERE delete_time IS NULL ORDER BY auto_termination_minutes DESC NULLS FIRST
Column names here are from memory and may differ in your workspace, so check the table's schema first. Run it against a small copy, as explained on the system tables page.
Checked
What was checked, and what was not
Checked against Databricks documentation on 21 November 2026:
- SQL warehouse types and creating a warehouse (size range, scaling defaults, auto-stop, API default type, quota)
- Warehouse sizing and scaling
- Compute configuration reference (autoscaling, spot and
first_on_demand, policies) and fleet instance types (families, local disk, GPU and Graviton limits) - Access modes and autoscaling for pipelines (and the advice to avoid autoscaling for streaming)
Not confirmed: the default auto-termination and default access mode for all-purpose compute, whether Photon is on by default on classic compute, the spot policy, channel and tags options on warehouses, which node families are best for the disk cache, the system.compute.clusters column names, and anything specific to Azure or Google Cloud, where machine names and defaults differ. Check each in your workspace.
FAQ
Questions people ask
Do classic SQL warehouses still exist?
Yes. The three types are serverless, pro and classic, and I found no deprecation notice. Serverless is the UI default where available, and classic or pro remain for custom networking, hybrid connections, or when serverless is not enabled.
Why might my warehouse be classic when I wanted serverless?
Warehouses created through the API or Terraform without the type set have defaulted to classic. Set the warehouse type and the serverless flag explicitly.
Should I turn on autoscaling for a streaming job cluster?
Databricks advises against it, because classic autoscaling is slow to shrink streaming clusters. Use a fixed size, or move the workload to a pipeline with enhanced autoscaling.
Can I use Graviton for everything?
No. It is documented as unsupported for Databricks SQL, declarative pipelines and Container Services, and Python UDFs need runtime 15.2 or later. Test plain batch jobs first.
Next
Where to go next
Related on this site: Which compute for which job · Biggest cost and performance levers · Tagging best practices · Cost and FinOps for the platform
All Databricks compute and cost pages · All data engineering · Databricks on this site