Home › Learn › Databricks compute and cost › Compute settings

Databricks compute and cost · data engineering

Databricks compute settings reference: SQL warehouse types, job clusters, node types, access modes and wrong settings

Every kind of compute in one map, then the settings for SQL warehouses (serverless, pro, classic), serverless and classic jobs, all-purpose clusters and node types, with the good value, the wrong value and what to check.

Cluster anatomyWarehouses and serverlessChoosing computeCompute settingsCost by workloadLogs and visibilityPhoton and cachesHow billing worksDividing the billSystem tablesSharing and regionsBackup architectureCost-saving hacks

Prices, defaults, limits and preview status change often, and dollar figures here are illustrations. Check the current Databricks documentation and price list before quoting a number.

The map

Every kind of compute, in one table

Three words that are not the same thing. Classic vs serverless says who runs the machines: you (in your cloud account) or Databricks (in its account). All-purpose vs jobs vs SQL warehouse says what the compute is for. Interactive is just a way of working, typing in a notebook, and is not a product. So an all-purpose cluster is a classic cluster used for interactive work. A job cluster is also classic, but it is not all-purpose. A serverless notebook is interactive, but it is not classic.

There are three families: SQL warehouses (for SQL and BI), jobs compute (for scheduled work) and all-purpose compute (for interactive work). Each exists in a classic form that runs in your own cloud account, and most have a serverless form that runs in Databricks' account.

ComputeClassic formServerless formUsed for
SQL warehouseClassic and Pro warehouses (VMs in your account)Serverless warehouseSQL editor, dashboards, Power BI, apps sending SQL
Jobs computeJob cluster, created for one run and deleted afterServerless jobsScheduled and triggered batch work
All-purpose computeAll-purpose cluster, in standard or dedicated access modeServerless notebooksInteractive development, exploration, ML
PipelinesClassic pipeline computeServerless pipelinesDeclarative bronze, silver and gold

Yes, classic warehouses still exist. I found no deprecation notice. The three warehouse types are serverless, pro and classic. For choosing between them, see choosing compute. This page is the settings reference: what each setting does, what to set and what people set wrongly.

Warehouses

SQL warehouse settings: serverless, pro and classic

Which type, and why classic still exists

ServerlessProClassic
Runs inDatabricks' accountYour cloud accountYour cloud account
Start-upSecondsSeveral minutes (typically about 4)Several minutes (typically about 4)
PhotonYesYesYes
Predictive IOYesYesNo
Intelligent Workload ManagementYesNoNo
ScalingAutomatic, within your minimum and maximumSlower to scale up and downSlower to scale up and down
Why choose itThe default choice when it is availableCustom networking, federation or hybrid connections, or serverless not availableBasic interactive exploration when neither other type is possible

The settings

SettingWhat it doesGood valueWrong value
Cluster sizeThe size of each cluster, from 2X-Small (one worker) to 4X-Large (256 workers). It sets how fast one query runs.Start small and raise it only if a query is slow or spills to disk. Databricks suggests starting a serverless warehouse larger and stepping down while testing.Leaving a large default for light queries, or raising size to fix a queue
Scaling: minimum and maximum clustersHow many clusters run at once. This sets concurrency. The default is one and one.Maximum sized to the real peak. Rule of thumb: one cluster per 10 concurrent queries.Maximum of 1 with many dashboard users, or a very high maximum with no cost alert
Auto stopMinutes of idle time before it stopsServerless: the default is 10 minutes, the UI minimum is 5, the API allows 1. Pro and classic: default 45, minimum 10. Shorten for ad-hoc use.Leaving 45 minutes on a pro or classic warehouse
Warehouse typeServerless, pro or classicServerless unless you need networking controlPro by habit, or classic by an API default
Statement timeoutStops a query that runs too long. The default is 2 days. Set at session or workspace level, with a warehouse-level setting in Beta.A limit that suits your longest legitimate queryLeaving the 2-day default
PermissionsWho can use or manage the warehouseUse for the team that owns it, manage for fewEveryone can manage
TagsAttach cost attribution labelsTeam, environment and owner on every warehouseNo tags, so the bill cannot be split
Default warehouseAn admin can set a workspace-wide default warehouse for ad-hoc queries (introduced in 2026). It works with all three types.Point it at a small serverless warehouseLeaving ad-hoc users on a large shared one

Not confirmed from the docs I could search: the Advanced options in the create form, which include the spot instance policy (pro and classic only), the release channel (current or preview) and tags. Open the form in your workspace and read each option's help text before relying on my description.

Classic vs pro

Classic SQL warehouse vs pro SQL warehouse, in detail

Both run on machines in your cloud account, both take several minutes to start (typically about 4), and both scale less quickly than serverless. The documented differences are small, and the price per DBU is not.

ClassicPro
Photon engineYesYes
Predictive IO (faster reads, fewer files scanned)NoYes
Intelligent Workload ManagementNoNo
Materialized views, streaming tables and query federationThe official pricing page lists these under Pro and Serverless. A third-party comparison says classic lacks them. I could not confirm classic's status, so check the docs.Yes
Illustrative rate per DBUabout $0.22about $0.55
Intended use, per DatabricksBasic interactive exploration when neither serverless nor pro is possibleWhen serverless is not available, or you need custom networking (for example to connect other services in your network, or hybrid and on-premises connections)

The settings are the same for both

Cluster size, minimum and maximum clusters, auto stop (default 45 minutes, minimum 10), warehouse type, tags, permissions, statement timeout, and the Advanced options. Spot instance policy applies only to the types that run on your machines, so it is the one setting that serverless does not have. I could not confirm its exact options from the docs I could search.

What decides it

FactorLeans toward
You only run basic interactive queries and want the lowest DBU rateClassic
Large scans where faster reads matter (Predictive IO)Pro
You need materialized views, streaming tables or query federation in the warehousePro (confirm classic first)
You need the warehouse inside your private networkPro
Serverless is available and your networking allows itNeither. Use serverless.
Burst use, many short sessionsNeither. Start-up of about 4 minutes and a 45-minute default stop make both costly.

Cost example

A Medium warehouse burns 24 DBU per hour. At the illustrative rates, the DBU part is $5.28 per hour on classic ($0.22 × 24), $13.20 on pro ($0.55 × 24) and $16.80 on serverless ($0.70 × 24). Classic and pro also bill the same cloud machines, so the VM part is added to both and the gap between them is only the DBU difference: $7.92 per hour, or about $697 over 88 busy hours. Serverless includes the machines and adds no idle tail beyond its short auto-stop. Pro is worth that extra DBU cost only when it gives you a feature you need.

Not confirmed: whether Python UDFs, materialized views and streaming tables work on classic warehouses today, and the exact spot policy options. The rates are illustrative third-party summaries. Check the warehouse types page and your price list.

Jobs

Jobs: serverless or classic job cluster

UseWhen
Serverless jobsPython or SQL tasks, no special needs, bursty or modest volume, you do not want to tune a cluster. Use Standard performance mode when a start-up of 4 to 6 minutes is fine.
Classic job clusterScala or RDD code, custom images or init scripts, custom networking, always-on streaming, runs longer than 7 days, specific machine types, or large steady work where spot pricing wins.

Serverless job settings

Performance mode, environment version and libraries, job timeout, retries, notifications, tags and a budget policy. There are no machine settings. See choosing compute.

Classic job cluster settings

SettingWhat it doesGood valueWrong value
Databricks Runtime versionSpark, Python and library versionsA current long-term-support version, pinnedAn old runtime that is out of support, or "latest" that changes under you
PhotonA faster engine at a higher DBU rateOn for SQL-heavy and large scans, after a testOn for Python UDF code, or never tested
Worker node typeThe machine family and size for workers (see the node types section)Match the bottleneck: memory for spill, compute for CPUThe largest type "to be safe"
Driver node typeThe machine for the driver. It can differ from workers.Smaller than workers unless you collect big resultsA big driver on every job, or a tiny one with collect()
Workers: fixed or autoscalingA fixed count, or a minimum and maximumAutoscaling for batch with uneven stages. Fixed size for continuous streaming.Autoscaling on a streaming job (Databricks advises against it, because it is slow to shrink), or autoscaling on a spark-submit job, which does not support it
Availability (spot)On-demand, spot, or spot with fallback. first_on_demand sets how many of the first nodes, starting with the driver, are on-demand.Spot with fallback, first_on_demand of 1 to keep the driver on-demandAll spot for a job that cannot restart, or all on-demand for large fault-tolerant batch
Access modeStandard (shared) or dedicated (single user or group)Standard for most workDedicated by default
Spark configurationOverrides Spark defaultsFew, documented and justifiedCopied lists from old blog posts that fight AQE
Tags and policyCost labels, and guardrails on what users can createRequired tags and a policy that caps sizeNo policy, so anyone can launch a huge cluster
Timeout and retriesStop a hung job, and rerun a failed oneA timeout and a small retry count (the timeout applies to each retry)No timeout, or max_retries of -1
Reusing a job cluster across tasksPays start-up once for several tasksShare it for similar tasksA separate cluster for every small task

Server types

Node types: which machine for which problem

On AWS, Databricks node types follow the EC2 families. Other clouds have equivalents with different names. Databricks also offers fleet instance types, which can draw from several similar machines to reduce capacity errors.

FamilyUse it whenNotes
General purpose (m-family, fleet: md-fleet)You are unsure, or the job is balancedA safe starting point
Memory optimized (r-family, fleet: rd-fleet, rgd-fleet)The Spark UI shows spill to disk, wide joins, or big aggregationsOften better than adding more general-purpose nodes
Compute optimized (c-family)The job is CPU-bound and does little shufflingToo little memory for large joins
Storage optimized (i-family)Read-heavy and interactive work that benefits from local SSD and the disk cacheI did not find a Databricks fleet type for this family
GPU (g and p families)Model training and some inferenceNot available as fleet types. Pick a specific GPU type, usually on dedicated access mode.
Graviton (Arm)Plain batch jobs, to cut the VM priceDocumented as unsupported for Databricks SQL, declarative pipelines and Container Services. Python UDFs need runtime 15.2 or later. Not offered as a fleet type.
Symptom in the Spark UITry
Spill to disk, out-of-memory errorsMemory optimized nodes, or fewer cores per node
CPU at 100 percent, little shuffleCompute optimized nodes
Repeated scans of the same dataNodes with local SSD, so the disk cache helps
Nodes below about 30 percent CPU and memoryFewer or smaller nodes
One slow task while others finishSkew. A different node type will not fix it.

All-purpose

All-purpose compute: access modes and the usual mistakes

MistakeWhy it hurtsFix
Auto-termination off or set to many hoursAn idle cluster bills all night and at weekendsA policy that forces a short auto-termination
A big cluster for a notebook that reads a small tablePays for unused coresSingle node, or serverless notebooks
Scheduled jobs left on an all-purpose clusterHigher DBU rateMove to jobs compute or serverless jobs
Dedicated mode by defaultFewer users per cluster and more clustersStandard unless you need RDD, GPU or R
No cluster policyAnyone can launch an expensive clusterPolicies that cap size, force tags and set auto-termination

Find the wrong settings in your own workspace

SELECT cluster_id, cluster_name, auto_termination_minutes, worker_node_type, min_autoscale_workers, max_autoscale_workers FROM system.compute.clusters WHERE delete_time IS NULL ORDER BY auto_termination_minutes DESC NULLS FIRST

Column names here are from memory and may differ in your workspace, so check the table's schema first. Run it against a small copy, as explained on the system tables page.

Checked

What was checked, and what was not

Checked against Databricks documentation on 21 November 2026:

Not confirmed: the default auto-termination and default access mode for all-purpose compute, whether Photon is on by default on classic compute, the spot policy, channel and tags options on warehouses, which node families are best for the disk cache, the system.compute.clusters column names, and anything specific to Azure or Google Cloud, where machine names and defaults differ. Check each in your workspace.

FAQ

Questions people ask

Do classic SQL warehouses still exist?

Yes. The three types are serverless, pro and classic, and I found no deprecation notice. Serverless is the UI default where available, and classic or pro remain for custom networking, hybrid connections, or when serverless is not enabled.

Why might my warehouse be classic when I wanted serverless?

Warehouses created through the API or Terraform without the type set have defaulted to classic. Set the warehouse type and the serverless flag explicitly.

Should I turn on autoscaling for a streaming job cluster?

Databricks advises against it, because classic autoscaling is slow to shrink streaming clusters. Use a fixed size, or move the workload to a pipeline with enhanced autoscaling.

Can I use Graviton for everything?

No. It is documented as unsupported for Databricks SQL, declarative pipelines and Container Services, and Python UDFs need runtime 15.2 or later. Test plain batch jobs first.

Next

Where to go next

Related on this site: Which compute for which job · Biggest cost and performance levers · Tagging best practices · Cost and FinOps for the platform

All Databricks compute and cost pages · All data engineering · Databricks on this site