Home › Learn › Databricks interview prep › Cost and performance levers
Databricks interview prep · data engineering
The biggest cost and performance levers
Ranked levers for Databricks cost and performance, how to find your own with system tables and the query profile, plus egress and ingress costs.
Defaults, limits and preview status change often. Check the current Databricks docs before quoting a number.
Measure first. Waste and data scanned are usually the biggest levers.
Cost
The biggest cost lever: stop paying for compute that does no useful work
| Rank | Lever | First action |
|---|---|---|
| 1 | Idle and always-on compute (interactive clusters left running, 24/7 streams that need only hourly freshness) | Auto-termination via policy; availableNow on a schedule |
| 2 | Wrong compute type (scheduled jobs on all-purpose clusters) | Job clusters or serverless jobs |
| 3 | Processing too much data (full reloads, wide scans) | Incremental loads; better layout |
| 4 | Oversized or poorly tuned clusters | Right-size from measured utilization |
| 5 | On-demand workers for fault-tolerant batch | Spot workers, on-demand driver |
| 6 | Slow code (skew, spill, UDFs) | Fix the top few expensive jobs |
| 7 | SQL warehouse size and auto-stop | Short auto-stop; serverless for bursty BI |
| 8 | Storage and egress | VACUUM; lifecycle rules; same-region compute |
Cost is price per hour times hours running. Tuning one query by 20 percent saves 20 percent of one job. Moving 50 jobs off all-purpose clusters saves a large part of the bill at once. Find your own lever first: query system.billing.usage with list prices, group by job, cluster and tag, and look at the top ten. Keep gains with cluster policies, budget policies, tags and spend alerts.
Performance
The biggest performance lever: read less data
| Rank | Lever | First action |
|---|---|---|
| 1 | Data skipping through file layout | Liquid clustering on filter and join columns; stats on those columns (first 32 by default) |
| 2 | Small files | OPTIMIZE, optimized writes, auto compaction, Predictive Optimization |
| 3 | Shuffle volume | Filter and select early, broadcast, aggregate before joining |
| 4 | Skew | Remove junk keys, AQE, isolate or salt |
| 5 | Photon | SQL-heavy workloads; measure cost per job |
| 6 | Python UDFs | Built-in functions or vectorized pandas UDFs |
| 7 | Non-incremental processing | Streaming tables, change data feed, incremental MERGE |
| 8 | Right-sized compute | Scale on spill and utilization, not guesses |
| 9 | Caching | Disk and result caches; avoid .cache() by habit |
| 10 | Statistics | ANALYZE or Predictive Optimization |
Query time is roughly data scanned times cost per byte. Layout cuts the first factor for every query, forever. In the query profile check files read versus pruned, task time spread, shuffle and spill size, and time outside Photon.
Network
Egress and ingress
Data moving in is generally free; moving out is charged per GB. Full tables and hidden costs: Cheapest way to move data in and out.
Practice
Questions to rehearse
- How would you find your biggest cost lever?
- Why are idle clusters usually the biggest waste?
- A query is slow. In what order do you look?
- How do you lock in a saving so it does not drift back?
- How would you reduce egress between two regions?
- Auto Loader or COPY INTO for cheap ingestion?
FAQ
Questions people ask
What is the quickest cost win?
Auto-termination and moving scheduled work off all-purpose clusters. Both are settings, not code.
Is ingress free?
The transfer into the cloud usually is. Source egress, network path, compute and requests are not.
Next
Where to go next
All Databricks interview topics · All data engineering · Databricks on this site