Home › Learn › Databricks interview prep › Cost and performance levers

Databricks interview prep · data engineering

The biggest cost and performance levers

Ranked levers for Databricks cost and performance, how to find your own with system tables and the query profile, plus egress and ingress costs.

Maintenance commandsRow-level securityCompute choiceJoinsSkew and shuffleAQE, Photon, POCost and performance leversDeclarative and semanticAsset BundlesLakeflow ConnectUniFormVector Searchdbt and ODCSEgress and ingressMaintenance in picturesClustering and compactionMaintenance strategySlow writesBackup and recoveryTaggingQuestion bank

Defaults, limits and preview status change often. Check the current Databricks docs before quoting a number.

Measure with systemtables and the queryprofileFix the biggest itemLock it in with policiesand tagsRepeat

Measure first. Waste and data scanned are usually the biggest levers.

Cost

The biggest cost lever: stop paying for compute that does no useful work

RankLeverFirst action
1Idle and always-on compute (interactive clusters left running, 24/7 streams that need only hourly freshness)Auto-termination via policy; availableNow on a schedule
2Wrong compute type (scheduled jobs on all-purpose clusters)Job clusters or serverless jobs
3Processing too much data (full reloads, wide scans)Incremental loads; better layout
4Oversized or poorly tuned clustersRight-size from measured utilization
5On-demand workers for fault-tolerant batchSpot workers, on-demand driver
6Slow code (skew, spill, UDFs)Fix the top few expensive jobs
7SQL warehouse size and auto-stopShort auto-stop; serverless for bursty BI
8Storage and egressVACUUM; lifecycle rules; same-region compute

Cost is price per hour times hours running. Tuning one query by 20 percent saves 20 percent of one job. Moving 50 jobs off all-purpose clusters saves a large part of the bill at once. Find your own lever first: query system.billing.usage with list prices, group by job, cluster and tag, and look at the top ten. Keep gains with cluster policies, budget policies, tags and spend alerts.

Performance

The biggest performance lever: read less data

RankLeverFirst action
1Data skipping through file layoutLiquid clustering on filter and join columns; stats on those columns (first 32 by default)
2Small filesOPTIMIZE, optimized writes, auto compaction, Predictive Optimization
3Shuffle volumeFilter and select early, broadcast, aggregate before joining
4SkewRemove junk keys, AQE, isolate or salt
5PhotonSQL-heavy workloads; measure cost per job
6Python UDFsBuilt-in functions or vectorized pandas UDFs
7Non-incremental processingStreaming tables, change data feed, incremental MERGE
8Right-sized computeScale on spill and utilization, not guesses
9CachingDisk and result caches; avoid .cache() by habit
10StatisticsANALYZE or Predictive Optimization

Query time is roughly data scanned times cost per byte. Layout cuts the first factor for every query, forever. In the query profile check files read versus pruned, task time spread, shuffle and spill size, and time outside Photon.

Network

Egress and ingress

Data moving in is generally free; moving out is charged per GB. Full tables and hidden costs: Cheapest way to move data in and out.

Practice

Questions to rehearse

  1. How would you find your biggest cost lever?
  2. Why are idle clusters usually the biggest waste?
  3. A query is slow. In what order do you look?
  4. How do you lock in a saving so it does not drift back?
  5. How would you reduce egress between two regions?
  6. Auto Loader or COPY INTO for cheap ingestion?
In an interview, say: I start by measuring, in system tables and the query profile. The biggest cost lever is usually idle or mis-typed compute; the biggest performance lever is reading less data. Hardware is the last lever, because it makes a bad plan faster without making it smaller.

FAQ

Questions people ask

What is the quickest cost win?

Auto-termination and moving scheduled work off all-purpose clusters. Both are settings, not code.

Is ingress free?

The transfer into the cloud usually is. Source egress, network path, compute and requests are not.

Next

Where to go next

All Databricks interview topics · All data engineering · Databricks on this site