Home › Learn › Databricks compute and cost

Databricks compute and cost · data engineering

Databricks compute and cost, explained end to end

Read it in order once, then use it as a reference. Plain words first, a drawing for every mechanism, and a Pursue or Drop verdict wherever there is a choice.

Prices, defaults, limits and preview status change often, and dollar figures are illustrations. Check the current Databricks documentation before quoting a number.

Start here

Why it feels confusing: four separate questions

People mix up four separate questions. Keep them apart and most of the confusion goes away.

QuestionPossible answersWhere it is explained
Who owns the machines?You (classic) or Databricks (serverless)Two planes
What kind of work runs on it?Notebooks and code (clusters) or SQL only (SQL warehouse)The family, warehouses and serverless
How fast does it run the work?Plain Spark engine or Photon enginePhoton
How long does it live?Stays up for a team (interactive) or exists for one run (job)Classic clusters

"SQL Warehouse", "Serverless" and "Photon" are not three competing things. A SQL warehouse is a type of compute. Serverless is who owns the machines. Photon is the engine inside. A serverless SQL warehouse is all three at once.

Databricks cluster architecture: planes, driver, workers, memory and machine types

The control plane and compute plane, the driver and worker anatomy of every cluster, all-purpose and job clusters and pools, what lives in RAM and on local disk, and the five machine families.

SQL warehouses, serverless and who owns it: classic, pro, serverless, prod and dev, BU1 and BU2

Classic, pro and serverless SQL warehouses compared, serverless notebooks and jobs, performance versus standard mode, who owns serverless (account, workspace, prod and dev, business units), and which to pick.

Choosing Databricks compute: classic, serverless, SQL warehouses, users, batch and live access, settings and mistakes

How many kinds of serverless there are, what you can and cannot configure on each, the factors that decide the choice (users, concurrency, latency, batch versus live, language, networking), scenarios with verdicts, sizing, common mistakes and what to watch out for.

Databricks compute settings reference: SQL warehouse types, job clusters, node types, access modes and wrong settings

Every kind of compute in one map, then the settings for SQL warehouses (serverless, pro, classic), serverless and classic jobs, all-purpose clusters and node types, with the good value, the wrong value and what to check.

Databricks cost map: every workload type, every compute type, DBU rates, the mapping and what goes wrong

One table of workload types, one of compute types with illustrative DBU rates, and the mapping between them with worked monthly examples, the wrong-compute multipliers and a monthly check.

Why serverless feels opaque: what Databricks logs, how to extract it programmatically and the cheapest way to read it

What you lose and what you keep on serverless, every source of run, query, billing, audit and lineage information with retention, how to extract it by SQL and REST API, and the cheapest way to do it.

Photon and every Databricks cache: result, Spark, disk, page and metadata caches

How Photon replaces Spark's JVM operators with a vectorized C++ engine, when it helps, and which cache each compute type supports: result cache, Spark cache, disk cache, OS page cache, Delta metadata cache.

How Databricks billing works: DBUs, two invoices, SKUs, autoscaling and billing tracing

The DBU formula, classic versus serverless invoices, illustrative rates and worked examples, SKU families, what autoscaling does to cost and what you can see, and why people say they need classic for billing tracing.

Dividing the Databricks bill: the chargeback architecture

Force tags, record usage, roll it up: the architecture for charging Databricks cost to business units and environments, the four workspace layouts compared, and the per-query gap.

Databricks system tables: contents, why reading them costs money, copying them cheaply, and what is hidden

What the system schemas hold, why querying them always costs compute, the cheapest incremental copy, whether Lakebase or UniForm would help, and the information Databricks does not show.

Delta Sharing inside, and how to avoid cross-region cost on Databricks

How Delta Sharing works internally (signed file links read straight from storage), when it helps with cost, why cross-region transfer is charged and the seven places it hides in Databricks.

Backup and recovery architecture for Databricks: the safe, clean design on any cloud

Time travel is not a backup. Three protections for three disasters: fast undo, an immutable backup in a separate account, and a rebuildable second region, with the design rules and a failure-to-tool table.

Databricks cost-saving hacks: the biggest levers, hidden tricks and checked settings

Find where the money goes, then the compute, warehouse, pipeline, data and cloud-side changes that cut a Databricks bill, each with a Pursue or Drop verdict and the limits that were checked against current documentation.

Already on this site

Related pages

Which compute for which job · AQE, Photon, Predictive Optimization · Biggest cost and performance levers · Tagging best practices · Cheapest way to move data in and out · Backup and recovery on Azure · UniForm · Who maintains your tables · Cost and FinOps for the platform · Databricks cost calculator

Check these

Bits to double-check against current docs

Databricks changes compute features often, and I wrote this from knowledge rather than a live docs check. These are the places I would verify before relying on a number or a limit:

  • Exact start-up times and the DBU price multipliers for Photon, Pro and serverless.
  • Which Spark cache calls and which languages serverless notebooks and jobs support today.
  • Whether disk cache is on by default for a given instance type on your cloud.
  • Whether the serverless-specific Predictive IO and workload management features differ for notebooks and jobs.
  • Remote result cache length (I wrote up to 24 hours).
  • Every dollar rate and the DBU-per-size table in the billing page (illustrative only), and the exact SKU names.
  • Serverless budget policy details and which serverless products they cover, and exact system table names, columns and retention.
  • Whether your contract discount is reflected anywhere in system tables.
  • Lakebase synced-table behaviour and pricing, and whether system tables can be read as a stream.
  • Cloud cross-region and cross-zone transfer rates for your provider, and the current UNDROP window.
  • Delta Sharing egress options (such as replicas or zero-egress storage) for your cloud.

The core structure, though, is stable: driver plus workers, control plane versus compute plane, Photon as a swap-in engine, and the cache layers from RAM to SSD to cloud storage.