Home › Learn › Databricks compute and cost › Cluster anatomy
Databricks compute and cost · data engineering
Databricks cluster architecture: planes, driver, workers, memory and machine types
The control plane and compute plane, the driver and worker anatomy of every cluster, all-purpose and job clusters and pools, what lives in RAM and on local disk, and the five machine families.
Prices, defaults, limits and preview status change often, and dollar figures here are illustrations. Check the current Databricks documentation and price list before quoting a number.
Cluster architecture
Two planes: where the machines live
Databricks splits itself into a control plane (the brain: web app, job scheduler, notebooks, cluster manager, Unity Catalog) and a compute plane (the muscle: the virtual machines that run Spark). Your data stays in your own cloud storage the whole time.
Rule of thumb. Classic means you pick machine types, versions and network details, and you wait for VMs to boot. Serverless means you pick almost nothing, Databricks tunes it, and it starts fast. You trade control for convenience.
Cluster architecture
Anatomy of one Spark cluster
Every cluster, whatever its name, is the same shape: one driver that plans and coordinates, and many workers that do the real work. A worker runs an executor, and an executor has one slot for each CPU core. A slot runs one task (one small piece of your query) at a time.
Cluster architecture
The whole family at a glance
| Compute type | Machines owned by | What you run | Life span | Engine | Starts in (rough) |
|---|---|---|---|---|---|
| All-purpose (interactive) cluster | You | Notebooks, Python, Scala, R, SQL, ML | Until auto-terminate | Spark, Photon optional | 3 to 8 min (seconds with a pool) |
| Job cluster | You | Scheduled jobs and pipelines | One run, then deleted | Spark, Photon optional | 3 to 8 min (seconds with a pool) |
| Pool | You | Nothing itself. Holds idle VMs for clusters. | Always on, holds spare VMs | none | Makes clusters start fast |
| Classic SQL warehouse | You | SQL only | Until auto-stop | Photon always on | Several minutes |
| Pro SQL warehouse | You | SQL only | Until auto-stop | Photon + Predictive IO | A few minutes |
| Serverless SQL warehouse | Databricks | SQL only | Until auto-stop (short) | Photon + Predictive IO + smart scaling | Seconds |
| Serverless notebook / job compute | Databricks | Python and SQL notebooks, jobs (check docs for other languages) | Per session or run | Managed Spark, Photon | Seconds in performance mode |
Also in the family, but built from the same parts: single-node clusters (driver only, for small work), GPU and ML runtime clusters (for deep learning), and pipeline compute for Lakeflow / Delta Live Tables (a job-style cluster with smarter autoscaling, or serverless).
Cluster architecture
Classic clusters in detail
All-purpose (interactive) cluster
The one you create from the Compute page and attach notebooks to. Several people can share it. It stays on until it hits the auto-terminate timer, so it is the easiest compute to forget and overpay for. It costs more per hour than a job cluster for the same machines.
- Access mode decides sharing and security: standard (shared by users, isolated) or dedicated (one user or group, full feature set).
- Runtime is a version of Databricks Runtime (DBR). It bundles a Spark version, Delta, Python and libraries. You choose it and you must upgrade it.
- Autoscaling adds workers between your minimum and maximum. Scaling up needs VM boot time, so it reacts in minutes.
- Spot instances use cheap interruptible VMs for workers, with fallback to on-demand. The driver should stay on-demand.
- Init scripts, custom libraries, Spark configs are all yours to change. This freedom is what serverless takes away.
Job cluster
The cluster is created when a job starts and deleted when it ends. Nobody shares it, nothing is left running, and the hourly rate is lower than all-purpose. This is the default choice for production pipelines on classic compute.
Pools
A pool keeps a set of already-booted idle VMs. A cluster that asks for machines from a pool skips the cloud's VM boot step, so it starts in seconds instead of minutes. You pay the cloud provider for idle pool VMs, but not Databricks DBUs for them.
Cluster architecture
Inside one worker: memory and disk
This is the picture that explains caches, spills and why machine type matters. Read it from the top (fastest, smallest) to the bottom (slowest, biggest).
Cluster architecture
Machine types: compute, memory, storage, general, GPU
On classic compute you pick the VM family. Serverless picks for you. The names differ by cloud, but the five families are the same idea everywhere.
| Family | Shape | Examples (AWS / Azure) | Pick it when | Verdict |
|---|---|---|---|---|
| General purpose | Balanced CPU and RAM | m5 / D-series | You do not know yet, mixed workloads, dev and test | Pursue as the starting point |
| Compute optimized | More CPU per GB of RAM | c5 / F-series | Streaming, light transforms, CPU-bound functions | Pursue for CPU-bound jobs only |
| Memory optimized | More RAM per core | r5 / E-series | Big joins, wide aggregations, heavy Spark cache, skewed data, spill problems | Pursue when you see spill in the Spark UI |
| Storage optimized | Large fast local NVMe disk | i3, i4i, m5d / L-series | Repeated reads of the same tables, big shuffles, disk cache benefit, interactive analysis | Pursue for read-heavy and interactive clusters |
| GPU accelerated | GPU plus CPU | g5, p4 / NC, ND-series | Deep learning, model training, embeddings | Pursue for ML only. Drop for SQL and ETL. |
A useful sentence to remember: compute optimized is about CPU, memory optimized is about RAM, storage optimized is about local disk. Photon plus disk cache loves the storage optimized family because scans hit the local SSD instead of cloud storage.
FAQ
Questions people ask
What is the difference between the driver and a worker?
The driver plans the query, cuts it into tasks and collects small results. Workers run an executor that executes the tasks, one task per CPU core at a time. A driver that runs out of memory kills the whole application.
Which machine family should I pick?
General purpose when unsure, memory optimized when the Spark UI shows spill, storage optimized for read-heavy and interactive work that benefits from the SSD disk cache, compute optimized for CPU-bound jobs, GPU only for model training.
Why does a pool start clusters faster?
A pool keeps booted idle VMs, so a cluster skips the cloud's VM boot step. You still pay the cloud for idle pool VMs, but no Databricks DBUs for them.
Next
Where to go next
Related on this site: Which compute for which job · AQE, Photon, Predictive Optimization
All Databricks compute and cost pages · All data engineering · Databricks on this site