Home › Learn › Databricks interview prep › Compute choice

Databricks interview prep · data engineering

Which compute for which job

All-purpose, job, serverless, SQL warehouse and pool compute: when to use each, serverless settings, the best setup for streaming, and driver memory problems.

Maintenance commandsRow-level securityCompute choiceJoinsSkew and shuffleAQE, Photon, POCost and performance leversDeclarative and semanticAsset BundlesLakeflow ConnectUniFormVector Searchdbt and ODCSEgress and ingressMaintenance in picturesClustering and compactionMaintenance strategySlow writesBackup and recoveryTaggingQuestion bank

Defaults, limits and preview status change often. Check the current Databricks docs before quoting a number.

Explore: serverless notebook orsmall shared clusterRun production: job cluster orserverless jobServe dashboards: serverless SQLwarehouse

Develop interactively, run production on job compute, serve BI from a warehouse.

The choices

Compute families

ComputeUse it whenAvoid it when
All-purpose (interactive) clusterExploring, debugging, ad-hoc work shared by people.Scheduled jobs: higher cost per DBU, and idle time wastes money.
Job clusterScheduled batch ETL; you need specific libraries, Spark configs or init scripts.Quick experiments: startup takes minutes.
Serverless notebooks and jobsNew batch work, short or bursty jobs, teams that do not want to tune clusters.Init scripts, unusual libraries, GPUs, tight per-node cost control (check current limits).
Serverless SQL warehouseDashboards, ad-hoc SQL, many concurrent users, bursty demand.Heavy Python or ML.
Pro or classic SQL warehouseSteady load, or serverless not permitted in your network.Bursty load: startup takes minutes.
Pipeline compute (declarative pipelines)Streaming tables and materialized views in Lakeflow pipelines.Anything that is not a pipeline.
Instance poolClassic job clusters that must start fast, many short jobs.Low job volume: you pay for idle pool VMs.
Single nodeSmall data, pandas, scikit-learn.Anything that needs distributed Spark.
GPU / ML runtime clusterTraining and fine-tuning.Plain ETL.

Sharing

Access modes

ModeMeaningUse for
Standard (was shared)Many users, isolated from each other, Unity Catalog security enforced natively.Teams sharing development compute.
Dedicated (was single user)One user or group; full language and library freedom. Fine-grained policies via serverless filtering.ML, R, custom libraries, jobs as a service principal.

Mode names have changed over time; check the current labels.

Serverless

What you can and cannot set

A setting people ask about

Performance mode (the “tier” setting)

I am not sure which screen you saw, so these are the settings I know of. Tell me where it appears and I will be specific.

WhereWhat it does
Serverless jobs and pipelines: performance mode (performance optimized or standard)Performance optimized starts quickly and costs more; standard accepts slower startup for a lower price. Use standard for overnight batch where start time does not matter, and performance optimized for jobs people wait for. Check current names and start times.
Azure workspace tier (Standard or Premium)A workspace feature level, not a compute setting. Unity Catalog, row filters and many governance features need Premium. Standard has been phased out for new workspaces; check.
SQL warehouse type (serverless, pro, classic) and sizeType decides features and startup time; size decides power for one query; cluster count decides concurrency.
Classic clustersNo tier; you choose runtime, node type, Photon and autoscaling.

Classic knobs

Settings that matter

KnobGuidance
AutoscalingGood for variable load. For streaming and predictable jobs, fixed size is often steadier.
Spot instancesWorkers on fault-tolerant batch; keep the driver on-demand.
PhotonOften a win for SQL and DataFrame work; compare cost per job.
Node typeMemory-optimized for joins and caching; compute-optimized for CPU-bound work; storage-optimized for shuffle-heavy work.
RuntimeUse an LTS version for production.
Cluster policiesCap size, force auto-termination, require tags. The main cost guardrail.

Streaming

Best compute for streaming

Hard problem

Driver memory and its problems

The driver plans the work, schedules tasks and collects small results. If it dies, the whole application dies.

CauseFix
collect(), toPandas() on big dataKeep data distributed; write out instead; limit() before inspecting.
Large broadcastLower the broadcast threshold or remove the hint; filter the small side first.
Millions of files or tasksOPTIMIZE and compaction; fewer, larger partitions.
Huge Delta log without checkpointsMake sure checkpoints are written; avoid thousands of tiny commits.
Lineage growing in loopsWrite an intermediate table or checkpoint to cut the plan.
Big Python objects in closuresBroadcast variables or a join.
Many notebooks on one shared clusterFewer users per cluster; separate heavy users.
Actions inside Python for-loopsBatch the work into one job.

Symptoms: Java heap OOM on the driver, "driver is up but is not responsive, likely due to GC", maxResultSize exceeded, hangs between stages. Look at the cluster metrics page, the driver logs, and the code that returns data to the driver. A bigger driver is the last resort, not the first. Raising spark.driver.maxResultSize only hides the problem.

Practice

Questions to rehearse

  1. Why are all-purpose clusters dearer than job clusters, and what follows?
  2. When would you pick classic job clusters over serverless jobs today?
  3. Bigger warehouse versus more clusters: which fixes one slow query, and which fixes queueing?
  4. Why keep the driver on-demand when workers are spot?
  5. A job fails with driver OOM. What do you do?
  6. How would you cut the cost of a 24/7 stream?
In an interview, say: Develop on interactive or serverless, run production on job clusters or serverless jobs, serve BI from serverless SQL warehouses. For streams, ask first whether it needs to be always on.

FAQ

Questions people ask

Is serverless always cheaper?

No. It usually wins for bursty, short or unpredictable work. A tuned classic job on spot nodes can win for large, steady workloads. Compare cost per run from system tables.

Does a bigger driver make a job faster?

Usually not. It only helps when the bottleneck is on the driver: planning, collecting or broadcasting.

Next

Where to go next

All Databricks interview topics · All data engineering · Databricks on this site