Home › Learn › Databricks compute and cost › Cluster anatomy

Databricks compute and cost · data engineering

Databricks cluster architecture: planes, driver, workers, memory and machine types

The control plane and compute plane, the driver and worker anatomy of every cluster, all-purpose and job clusters and pools, what lives in RAM and on local disk, and the five machine families.

Cluster anatomyWarehouses and serverlessChoosing computeCompute settingsCost by workloadLogs and visibilityPhoton and cachesHow billing worksDividing the billSystem tablesSharing and regionsBackup architectureCost-saving hacks

Prices, defaults, limits and preview status change often, and dollar figures here are illustrations. Check the current Databricks documentation and price list before quoting a number.

Cluster architecture

Two planes: where the machines live

Databricks splits itself into a control plane (the brain: web app, job scheduler, notebooks, cluster manager, Unity Catalog) and a compute plane (the muscle: the virtual machines that run Spark). Your data stays in your own cloud storage the whole time.

CONTROL PLANE (Databricks account) Web app / notebooks Jobs scheduler Cluster manager Unity Catalog (rules) SQL gateway + queue CLASSIC COMPUTE PLANE (your account) All-purpose cluster (VMs) Job cluster (VMs) Classic SQL warehouse Pro SQL warehouse SERVERLESS PLANE (Databricks' account) Serverless SQL warehouse Serverless notebook Serverless job Warm pool of ready VMs YOUR storage: S3 / ADLS / GCS (Delta tables, Parquet files) "start me VMs" "run this query" read / write files read / write files
Orange is your account, blue is Databricks' account. In classic compute the VMs are created inside your cloud bill and your network. In serverless they come from a Databricks-managed pool that is already warm, which is why they start in seconds.

Rule of thumb. Classic means you pick machine types, versions and network details, and you wait for VMs to boot. Serverless means you pick almost nothing, Databricks tunes it, and it starts fast. You trade control for convenience.

Cluster architecture

Anatomy of one Spark cluster

Every cluster, whatever its name, is the same shape: one driver that plans and coordinates, and many workers that do the real work. A worker runs an executor, and an executor has one slot for each CPU core. A slot runs one task (one small piece of your query) at a time.

DRIVER node Spark driver: reads your code,builds a plan, cuts it into tasks Notebook session (REPL)Spark UI and logs Delta log cache (table metadata) Holds results of collect().Too small = out-of-memory. Never runs the heavy data work. WORKER 1 Executor (JVM) slotslot slotslot 4 cores = 4 slots Local SSDdisk cache + shuffle RAM: heap + off-heap WORKER 2 Executor (JVM) slotslot slotslot Local SSDdisk cache + shuffle WORKER 3 Added by autoscalingwhen tasks pile up,removed when idle. tasks workers swap shuffle data with each other (joins, group-bys)
Databricks runs one executor per worker VM. A cluster of 4 workers with 8 cores each runs 32 tasks at the same time. More cores means more parallelism. More RAM per core means fewer spills. A fast local SSD means faster shuffles and cache hits.

Cluster architecture

The whole family at a glance

Compute typeMachines owned byWhat you runLife spanEngineStarts in (rough)
All-purpose (interactive) clusterYouNotebooks, Python, Scala, R, SQL, MLUntil auto-terminateSpark, Photon optional3 to 8 min (seconds with a pool)
Job clusterYouScheduled jobs and pipelinesOne run, then deletedSpark, Photon optional3 to 8 min (seconds with a pool)
PoolYouNothing itself. Holds idle VMs for clusters.Always on, holds spare VMsnoneMakes clusters start fast
Classic SQL warehouseYouSQL onlyUntil auto-stopPhoton always onSeveral minutes
Pro SQL warehouseYouSQL onlyUntil auto-stopPhoton + Predictive IOA few minutes
Serverless SQL warehouseDatabricksSQL onlyUntil auto-stop (short)Photon + Predictive IO + smart scalingSeconds
Serverless notebook / job computeDatabricksPython and SQL notebooks, jobs (check docs for other languages)Per session or runManaged Spark, PhotonSeconds in performance mode

Also in the family, but built from the same parts: single-node clusters (driver only, for small work), GPU and ML runtime clusters (for deep learning), and pipeline compute for Lakeflow / Delta Live Tables (a job-style cluster with smarter autoscaling, or serverless).

Cluster architecture

Classic clusters in detail

All-purpose (interactive) cluster

The one you create from the Compute page and attach notebooks to. Several people can share it. It stays on until it hits the auto-terminate timer, so it is the easiest compute to forget and overpay for. It costs more per hour than a job cluster for the same machines.

Job cluster

The cluster is created when a job starts and deleted when it ends. Nobody shares it, nothing is left running, and the hourly rate is lower than all-purpose. This is the default choice for production pipelines on classic compute.

Pools

A pool keeps a set of already-booted idle VMs. A cluster that asks for machines from a pool skips the cloud's VM boot step, so it starts in seconds instead of minutes. You pay the cloud provider for idle pool VMs, but not Databricks DBUs for them.

Without a pool You press Start Cloud boots VMsslowest step Install Spark + libraries Ready With a pool You press Start Take idle VMs from poolalready booted Install Spark + libraries Ready, much sooner
A pool removes the VM boot step. Serverless goes further: it keeps the VMs warm and the software ready, so the whole path is a few seconds.

Cluster architecture

Inside one worker: memory and disk

This is the picture that explains caches, spills and why machine type matters. Read it from the top (fastest, smallest) to the bottom (slowest, biggest).

ONE WORKER VM RAM: JVM heapSpark execution (sorts, joins) and storage memory RAM: off-heapPhoton's working memory, native buffers RAM: OS and Databricks agents RAM: OS page cache (all spare RAM) Local SSD / NVMeDisk cache(copies of data files) Local SSD / NVMeShuffle filesand spill to disk Cloud storage (S3/ADLS/GCS): big, cheap, slowest cache miss goes here Who uses each layer df.cache() / persist(): JVM heap (spills to disk if full) Photon: off-heap memory Spark execution: heap, spills to local SSD Disk cache: local SSD, faster than cloud reads Shuffles: written to local SSD, read by peers OS page cache: free, automatic, shared by all What goes wrong Not enough RAM per core: joins spill to SSD, slow. Slow disk: shuffles and cache reads drag. Cache too big for SSD: old entries are evicted. Skew: one slot does all the work, others sit idle.
Every optimisation is a trade among these layers. More RAM per core avoids spills. A faster local SSD speeds up shuffles and the disk cache. Photon changes which RAM region does the work.

Cluster architecture

Machine types: compute, memory, storage, general, GPU

On classic compute you pick the VM family. Serverless picks for you. The names differ by cloud, but the five families are the same idea everywhere.

FamilyShapeExamples (AWS / Azure)Pick it whenVerdict
General purposeBalanced CPU and RAMm5 / D-seriesYou do not know yet, mixed workloads, dev and testPursue as the starting point
Compute optimizedMore CPU per GB of RAMc5 / F-seriesStreaming, light transforms, CPU-bound functionsPursue for CPU-bound jobs only
Memory optimizedMore RAM per corer5 / E-seriesBig joins, wide aggregations, heavy Spark cache, skewed data, spill problemsPursue when you see spill in the Spark UI
Storage optimizedLarge fast local NVMe diski3, i4i, m5d / L-seriesRepeated reads of the same tables, big shuffles, disk cache benefit, interactive analysisPursue for read-heavy and interactive clusters
GPU acceleratedGPU plus CPUg5, p4 / NC, ND-seriesDeep learning, model training, embeddingsPursue for ML only. Drop for SQL and ETL.

A useful sentence to remember: compute optimized is about CPU, memory optimized is about RAM, storage optimized is about local disk. Photon plus disk cache loves the storage optimized family because scans hit the local SSD instead of cloud storage.

FAQ

Questions people ask

What is the difference between the driver and a worker?

The driver plans the query, cuts it into tasks and collects small results. Workers run an executor that executes the tasks, one task per CPU core at a time. A driver that runs out of memory kills the whole application.

Which machine family should I pick?

General purpose when unsure, memory optimized when the Spark UI shows spill, storage optimized for read-heavy and interactive work that benefits from the SSD disk cache, compute optimized for CPU-bound jobs, GPU only for model training.

Why does a pool start clusters faster?

A pool keeps booted idle VMs, so a cluster skips the cloud's VM boot step. You still pay the cloud for idle pool VMs, but no Databricks DBUs for them.

Next

Where to go next

Related on this site: Which compute for which job · AQE, Photon, Predictive Optimization

All Databricks compute and cost pages · All data engineering · Databricks on this site