Home › Learn › Databricks interview prep › Compute choice
Databricks interview prep · data engineering
Which compute for which job
All-purpose, job, serverless, SQL warehouse and pool compute: when to use each, serverless settings, the best setup for streaming, and driver memory problems.
Defaults, limits and preview status change often. Check the current Databricks docs before quoting a number.
Develop interactively, run production on job compute, serve BI from a warehouse.
The choices
Compute families
| Compute | Use it when | Avoid it when |
|---|---|---|
| All-purpose (interactive) cluster | Exploring, debugging, ad-hoc work shared by people. | Scheduled jobs: higher cost per DBU, and idle time wastes money. |
| Job cluster | Scheduled batch ETL; you need specific libraries, Spark configs or init scripts. | Quick experiments: startup takes minutes. |
| Serverless notebooks and jobs | New batch work, short or bursty jobs, teams that do not want to tune clusters. | Init scripts, unusual libraries, GPUs, tight per-node cost control (check current limits). |
| Serverless SQL warehouse | Dashboards, ad-hoc SQL, many concurrent users, bursty demand. | Heavy Python or ML. |
| Pro or classic SQL warehouse | Steady load, or serverless not permitted in your network. | Bursty load: startup takes minutes. |
| Pipeline compute (declarative pipelines) | Streaming tables and materialized views in Lakeflow pipelines. | Anything that is not a pipeline. |
| Instance pool | Classic job clusters that must start fast, many short jobs. | Low job volume: you pay for idle pool VMs. |
| Single node | Small data, pandas, scikit-learn. | Anything that needs distributed Spark. |
| GPU / ML runtime cluster | Training and fine-tuning. | Plain ETL. |
Sharing
Access modes
| Mode | Meaning | Use for |
|---|---|---|
| Standard (was shared) | Many users, isolated from each other, Unity Catalog security enforced natively. | Teams sharing development compute. |
| Dedicated (was single user) | One user or group; full language and library freedom. Fine-grained policies via serverless filtering. | ML, R, custom libraries, jobs as a service principal. |
Mode names have changed over time; check the current labels.
Serverless
What you can and cannot set
- Can: performance mode for jobs (fast start versus lower cost; check current names), the environment version and libraries, budget policies and tags, warehouse size and auto-stop, network connectivity configuration.
- Cannot (generally): choose the node type, run init scripts, set most Spark configs, SSH in. This list shrinks over time, so verify.
A setting people ask about
Performance mode (the “tier” setting)
I am not sure which screen you saw, so these are the settings I know of. Tell me where it appears and I will be specific.
| Where | What it does |
|---|---|
| Serverless jobs and pipelines: performance mode (performance optimized or standard) | Performance optimized starts quickly and costs more; standard accepts slower startup for a lower price. Use standard for overnight batch where start time does not matter, and performance optimized for jobs people wait for. Check current names and start times. |
| Azure workspace tier (Standard or Premium) | A workspace feature level, not a compute setting. Unity Catalog, row filters and many governance features need Premium. Standard has been phased out for new workspaces; check. |
| SQL warehouse type (serverless, pro, classic) and size | Type decides features and startup time; size decides power for one query; cluster count decides concurrency. |
| Classic clusters | No tier; you choose runtime, node type, Photon and autoscaling. |
Classic knobs
Settings that matter
| Knob | Guidance |
|---|---|
| Autoscaling | Good for variable load. For streaming and predictable jobs, fixed size is often steadier. |
| Spot instances | Workers on fault-tolerant batch; keep the driver on-demand. |
| Photon | Often a win for SQL and DataFrame work; compare cost per job. |
| Node type | Memory-optimized for joins and caching; compute-optimized for CPU-bound work; storage-optimized for shuffle-heavy work. |
| Runtime | Use an LTS version for production. |
| Cluster policies | Cap size, force auto-termination, require tags. The main cost guardrail. |
Streaming
Best compute for streaming
- Always-on stream: a job cluster (not all-purpose), fixed or tightly bounded size, mostly on-demand nodes, memory-optimized if stateful, RocksDB state store, an LTS runtime.
- Autoscaling reacts slowly for streams and moves state when nodes leave; size for peak and test.
- If freshness of minutes or hours is fine, do not run a 24/7 stream. Use
trigger(availableNow=True)on a schedule. This is usually the biggest streaming saving. - Streams inside declarative pipelines use pipeline compute; serverless pipelines manage scaling.
Hard problem
Driver memory and its problems
The driver plans the work, schedules tasks and collects small results. If it dies, the whole application dies.
| Cause | Fix |
|---|---|
| collect(), toPandas() on big data | Keep data distributed; write out instead; limit() before inspecting. |
| Large broadcast | Lower the broadcast threshold or remove the hint; filter the small side first. |
| Millions of files or tasks | OPTIMIZE and compaction; fewer, larger partitions. |
| Huge Delta log without checkpoints | Make sure checkpoints are written; avoid thousands of tiny commits. |
| Lineage growing in loops | Write an intermediate table or checkpoint to cut the plan. |
| Big Python objects in closures | Broadcast variables or a join. |
| Many notebooks on one shared cluster | Fewer users per cluster; separate heavy users. |
| Actions inside Python for-loops | Batch the work into one job. |
Symptoms: Java heap OOM on the driver, "driver is up but is not responsive, likely due to GC", maxResultSize exceeded, hangs between stages. Look at the cluster metrics page, the driver logs, and the code that returns data to the driver. A bigger driver is the last resort, not the first. Raising spark.driver.maxResultSize only hides the problem.
Practice
Questions to rehearse
- Why are all-purpose clusters dearer than job clusters, and what follows?
- When would you pick classic job clusters over serverless jobs today?
- Bigger warehouse versus more clusters: which fixes one slow query, and which fixes queueing?
- Why keep the driver on-demand when workers are spot?
- A job fails with driver OOM. What do you do?
- How would you cut the cost of a 24/7 stream?
FAQ
Questions people ask
Is serverless always cheaper?
No. It usually wins for bursty, short or unpredictable work. A tuned classic job on spot nodes can win for large, steady workloads. Compare cost per run from system tables.
Does a bigger driver make a job faster?
Usually not. It only helps when the bottleneck is on the driver: planning, collecting or broadcasting.
Next
Where to go next
All Databricks interview topics · All data engineering · Databricks on this site