Home › Learn › Databricks compute and cost › Choosing compute

Databricks compute and cost · data engineering

Choosing Databricks compute: classic, serverless, SQL warehouses, users, batch and live access, settings and mistakes

How many kinds of serverless there are, what you can and cannot configure on each, the factors that decide the choice (users, concurrency, latency, batch versus live, language, networking), scenarios with verdicts, sizing, common mistakes and what to watch out for.

Cluster anatomyWarehouses and serverlessChoosing computeCompute settingsCost by workloadLogs and visibilityPhoton and cachesHow billing worksDividing the billSystem tablesSharing and regionsBackup architectureCost-saving hacks

Prices, defaults, limits and preview status change often, and dollar figures here are illustrations. Check the current Databricks documentation and price list before quoting a number.

Start here

How many kinds of serverless are there?

"Serverless" is not one thing. Databricks uses the word for several separate products, and each has different settings. For your own code and SQL there are four. Other serverless features are billed and configured separately.

KindWhat it runsWho uses it
1. Serverless SQL warehouseSQL queries from the SQL editor, dashboards, Power BI and other BI toolsAnalysts, BI tools, apps that send SQL
2. Serverless notebooksPython and SQL in an interactive notebookEngineers and data scientists developing
3. Serverless jobsScheduled or triggered tasks in Lakeflow JobsProduction batch pipelines
4. Serverless pipelinesLakeflow Spark Declarative PipelinesDeclarative bronze, silver and gold
Other serverless featuresModel Serving, Predictive Optimization, and others, each with its own endpoint or billing lineSpecific features. Check each one's page and price.

The first four share one idea: you do not pick machines, and Databricks runs the compute in its own account. Notebooks, jobs and pipelines all run on the same kind of serverless engine. A SQL warehouse is a separate product with its own sizing.

Enabling serverless for SQL warehouses is a separate step from enabling it for notebooks, jobs and pipelines, and availability depends on cloud and region. If a choice is greyed out in your workspace, ask your account admin before assuming it does not exist.

Configuration

What you can configure on each kind

KindYou can setYou cannot set
Serverless SQL warehouseCluster size (T-shirt size), the minimum and maximum number of clusters, auto-stop (5 minutes at the lowest in the UI, 1 minute through the API), tags, permissions, and a statement timeout at session or workspace levelMachine type, Photon on or off (it is always on), spot, init scripts, which VPC it runs in
Serverless notebookEnvironment version, base environment and libraries, and memory (standard is 16 GB, high is 32 GB, with a higher DBU rate for high memory)Machine type, number of workers, most Spark settings, compute policies
Serverless jobPerformance mode (performance optimized or standard), environment and libraries, job timeout, retries, notifications, tags and a budget policyMachine type, number of workers, autoscaling limits, Photon on or off, init scripts, custom images
Serverless pipelineServerless on or off, triggered or continuous mode, performance mode, and a budget policyMachines and worker counts

The pattern is simple: on serverless you set behaviour and limits (mode, timeout, memory, libraries, tags), and you give up machine-level control (node type, worker count, spot, init scripts, Photon). Only warehouses give you a sizing knob, because a warehouse is sized by T-shirt size and cluster count.

Two settings are worth knowing exactly: serverless jobs in Standard mode use fewer DBUs but typically take 4 to 6 minutes to start, and auto-stop on pro and classic warehouses defaults to 45 minutes with a 10-minute minimum, while serverless defaults to 10 minutes. The page on cost-saving hacks lists the rest.

Factors

The factors that decide the choice

FactorAskLeans toward
Workload shapeScheduled batch, interactive, live BI queries, or always-on streaming?Batch: jobs compute or serverless jobs. Interactive: serverless notebook. BI: SQL warehouse. Always-on streaming: classic job cluster.
ConcurrencyHow many queries run at the same moment, not how many people exist?Many concurrent queries: a warehouse with more clusters
LatencyDoes a user wait for the first result? Is a start-up of minutes acceptable?Seconds: serverless. Minutes fine: classic, or serverless Standard mode.
PredictabilitySteady 24 hours a day, or bursty?Bursty: serverless, because idle time is cheap. Steady and heavy: classic on spot or committed VMs can win.
Language and APIScala, R, RDD code, continuous streaming triggers?Classic or dedicated compute. Serverless restricts these.
Libraries and customisationCustom images, init scripts, special drivers?Classic
NetworkingMust the compute sit inside your private network, for example to reach a database that only has a private IP, or a firewall that allows only your own IP ranges?Serverless runs in Databricks' network, not yours. It can reach private resources only through a network connectivity configuration (private endpoints, stable IP ranges, outbound policies), which has limits: for example, firewall enablement is not supported for S3 or DynamoDB. If you cannot or may not set that up, use a pro warehouse or classic compute.
Run lengthCould a run exceed 7 days?Classic. Serverless stops runs after 7 days.
Cost attributionDo you need to trace each VM on the cloud bill?Classic, where tags reach the VMs. Serverless uses budget policies.
Team skillIs anyone available to tune clusters?No one: serverless
AvailabilityIs serverless enabled in your cloud and region?If not, pro warehouses and classic compute

Scenarios

Which compute for which job

ScenarioPickWhyVerdict
Scheduled ETL, nothing unusualServerless jobs, Standard mode if it can waitNo cluster to tune, and idle time is not billedPursue
Large steady ETL, Scala or RDD code, spot savings wantedClassic job clusterFull control and the lowest hourly rate on spotPursue
A few analysts running ad-hoc SQLOne small serverless SQL warehouse, short auto-stopStarts in seconds and costs almost nothing idlePursue
Dashboards for many users, Power BIServerless SQL warehouse with a higher maximum cluster count, one per business unitConcurrency scales out, and cost is attributablePursue
Live, always-on application queriesA warehouse sized for the peak, or a database designed for point lookupsWarehouses are for analytic queries. Many tiny key lookups are a different workload.Pursue with a load test
Notebook developmentServerless notebook, or standard all-purpose computeFast start for one userPursue
GPU, RDD, R, or privileged machine accessDedicated all-purpose computeThese need dedicated access modePursue
Always-on low-latency streamingClassic job cluster, continuousServerless does not support the continuous and processing-time triggersPursue
Custom networking or hybrid connectionsPro warehouse or classic computeThese run in your own cloud networkPursue
Declarative pipelinesServerless pipelines, triggered modeManaged scaling, no always-on clusterPursue
Scheduled production job on an all-purpose clusterMove it to jobs compute or serverless jobsAll-purpose carries a higher DBU rateDrop
One huge warehouse for every teamSeparate warehouses per business unitNo attribution, and teams slow each other downDrop

More detail on the warehouse types: SQL warehouses and serverless. Interview-style version: which compute for which job.

Sizing

Number of users, concurrency and performance

Count concurrent queries, not users. Ten users can mean two queries or two hundred. A dashboard with twelve tiles sends twelve queries each time it opens or refreshes.

Problem you seeWhat it meansFix
One query is slow, or spills to diskThe cluster is too small for that queryIncrease the size
Many queries wait in a queueToo few clusters for the concurrencyRaise the maximum number of clusters
Cost is too high, queries are fastThe warehouse is bigger than it needs to beStep down one size, as long as nothing spills or slows noticeably

Treat the 10-per-cluster figure as a planning rule, not a hard limit, and load test with your real dashboards.

Mistakes

Common mistakes

MistakeWhy it hurtsFix
Scheduled jobs left on all-purpose clustersHigher DBU rate, and the cluster often stays upJobs compute or serverless jobs
Making the warehouse bigger to fix a queueSize helps one query. Queues need more clusters.Raise the maximum cluster count
Dashboards refreshing every few minutesEach refresh keeps the warehouse awake all dayRefresh less often, and use result caching and precomputed tables
Choosing serverless for work it cannot runNo RDD APIs and no sparkContext, no continuous streaming triggers, a 7-day limitCheck the limits first, or use classic
One warehouse shared by every teamNo cost attribution, and noisy neighboursOne warehouse per business unit
Ignoring the queue metricUsers blame the tool while queries waitAlert on Peak Queued Queries
Long auto-stop on pro or classic warehousesThe default is 45 minutes of idle billingShorten it, or use serverless
Picking a pro warehouse when serverless is availableSlow start and no Intelligent Workload ManagementUse pro only for custom networking or when serverless is unavailable
Dedicated access mode by defaultDatabricks recommends standard for most work. Dedicated is for RDD, GPUs, R and privileged access.Start with standard
Turning Photon on or off without measuringIt raises the DBU rate, and helps only some workloadsTest your real job
Assuming serverless is always cheaperIts DBU rate is higher. It saves idle time and VM work, not every dollar.Compare total monthly cost for your usage pattern
Treating serverless as zero-effortA newer environment version can change behaviour, for example how errors surface under Spark ConnectPin the environment version and test before upgrading
No tags or budget policy on serverlessUsage cannot be split by team afterwardsSet a budget policy on every serverless workload

Watch out

What to watch out for

Quick chooser

Six questions, in order

  1. Is it SQL from a BI tool or editor? Use a serverless SQL warehouse. Use pro or classic only for custom networking or when serverless is unavailable.
  2. Is it interactive development? Use a serverless notebook, or standard all-purpose compute. Use dedicated for GPU, RDD or R.
  3. Is it a scheduled job? Use serverless jobs unless a limit blocks it. Use Standard mode if it can wait.
  4. Does it need Scala, RDD code, custom images, special networking, always-on streaming, or runs over 7 days? Use a classic job cluster.
  5. Is it large, steady and heavy? Price a classic cluster on spot against serverless for your real usage.
  6. Is it a declarative pipeline? Use serverless pipelines in triggered mode.

Worked decision

A shared dev setup for data engineers and data scientists, available any time

"Anytime they can query" does not mean "a cluster running 24 hours a day". It means a user can start work whenever they want and not wait long. Compute that starts in seconds, or restarts on demand, gives the same availability without paying for the empty hours.

OptionHow it worksIllustrative monthly costVerdict
A. One big shared cluster running 24 hours a day8 machines (16 DBU per hour) on all-purpose compute, always on730 h × 16 = 11,680 DBU × $0.55 = $6,424, plus VMs $2,243 = $8,667Drop. Most of those hours are empty.
B. One shared cluster that starts on demand and stops when idleStandard access mode, autoscaling from 2 to 8 machines, auto-terminate after 30 to 60 minutes. Users reconnect and it restarts.Active 12 h a day for 22 days at an average of 4 machines: 264 h × 8 DBU × $0.55 = $1,162, plus VMs $406 = $1,567Pursue when many users run light work and need Scala, RDD or libraries
C. Serverless notebooks, one per personNo cluster to wait for or share. Each person's session is isolated.You pay only while a session is active. At an assumed 2 DBU per active hour, 88 hours a month is 176 DBU × $0.75 = $132 per personPursue as the default for DE and DS
D. One small serverless SQL warehouse per team2X-Small (4 DBU per hour), starts in seconds, short auto-stop. Anyone can run SQL at any time.6 busy hours a day for 22 days: 132 h × 4 = 528 DBU × $0.70 = $370Pursue for ad-hoc SQL
E. Personal single-node classic clustersOne small machine per person, policy-limited, auto-terminate after 20 to 30 minutes88 active hours × 2 DBU × $0.55 = $97, plus VMs $34 = $131 per person, before idle tailsPursue for work serverless cannot do
F. An instance pool to speed up classic startsIdle VMs wait in a pool so a cluster starts fasterNo DBUs for idle pool machines, but your cloud bills them while they sitPursue only with a small minimum idle count

The cheapest effective setup

  1. Serverless notebooks for most DE and DS work (option C), because nothing runs, or bills, when nobody is working.
  2. One small serverless SQL warehouse per team (option D) for anyone who just wants to query.
  3. Personal single-node classic clusters or a shared standard cluster (E or B) only for what serverless cannot do: Scala or RDD code, custom images, GPUs, or private-network access.
  4. Run heavy or scheduled work as a job, not on the dev cluster. See the nightly ETL decision on the cost map.

Guardrails that matter more than the choice

What goes wrong

ProblemWhyFix
The "shared" cluster never stopsA forgotten loop, streaming query or auto-refresh counts as activity, so auto-termination never triggersPolicy, alerts, and no scheduled work on dev compute
One user's heavy query slows everyoneA shared cluster has shared memory and coresServerless or personal compute for heavy work, and a sampled dev catalog
Library conflicts on a shared clusterEveryone installs into one environmentNotebook-scoped libraries, or serverless environments per person
Standard mode blocks something people needStandard access mode has no R, and limits on RDD and Spark contextDedicated mode for those users only
Classic start-up kills the "anytime" promiseA cold classic cluster takes minutesServerless, or a small pool
Serverless cannot reach a private dev databaseIt runs in Databricks' networkA network connectivity configuration, or classic compute for that user

The figures use the same illustrative assumptions as the cost map: 2 DBU per machine per hour, $0.384 per machine per hour in cloud VMs, and third-party summaries of list rates. Serverless DBU use per session is an assumption.

Checked

What was checked, and what was not

These were checked against Databricks documentation on 21 November 2026:

Not confirmed: whether high memory is still labelled preview, the exact minimum and maximum cluster controls shown on a serverless warehouse in your workspace, the default auto-termination time of all-purpose clusters, which release channels apply to serverless warehouses and pipelines, and the full list of other serverless features. Check the settings page in your own workspace before you rely on any of them.

FAQ

Questions people ask

How many types of serverless does Databricks have?

Four for your own code and SQL: serverless SQL warehouses, serverless notebooks, serverless jobs and serverless pipelines. Other features such as Model Serving and Predictive Optimization are separate serverless products with their own settings and billing.

What can I configure on serverless jobs?

Performance mode, the environment version and libraries, the job timeout and retries, notifications, tags and a budget policy. You cannot choose machine types, worker counts, Photon or init scripts.

A warehouse is slow. Do I make it bigger or add clusters?

Make it bigger when a single query is slow or spills to disk. Add clusters when queries wait in a queue. Size helps one query, clusters help concurrency.

When should I use classic compute instead of serverless?

When you need Scala or RDD code, custom images or init scripts, custom networking, always-on low-latency streaming, runs longer than 7 days, or the lowest hourly price on spot for large steady work.

Next

Where to go next

Related on this site: Which compute for which job · Biggest cost and performance levers · Tagging best practices · Cost and FinOps for the platform

All Databricks compute and cost pages · All data engineering · Databricks on this site