Your Databricks bill has a leak: 10 cost mistakes we keep seeing

A Databricks bill rarely jumps because of one big decision. It leaks through ten small holes: the wrong kind of compute, clusters left running, tiny files, one slow task, Python where Spark would do, and no tags to say who spent what. This note shows how to spot each one, the fix in plain words, and a ten-minute weekly check.

The finance team sends a message on the first of the month: the Databricks bill went up again, can someone explain? Nobody can. No single person made a big, expensive decision. The money left through a lot of small holes at once.

We see the same holes again and again. Most of them are not exotic tuning problems. They are a cluster nobody turned off, a job running on the wrong kind of compute, or a table made of thousands of tiny files.

This note goes through the ten we see most. For each one: how to spot it, and the fix.

nobody decided to spend it; it just leaks outyour budgetidle clustersthe wrong kind of computewarehouses left onno cost tagssmall filesone slow task (skew)Python UDFscollect() on big dataplus two more: oversized clusters and the wrong number of partitions
The bill doesn't jump. It leaks, through many small holes at once.

Where the money actually goes

On Databricks you pay mostly for compute: machines running, measured in DBUs (Databricks Units, the unit the price list is written in) plus the cloud provider's charge for the virtual machines. Storage is usually a small part of the bill. So the first question is always the same: were the machines working, or just sitting there?

Friday 18:00 to Monday 09:00: 63 hours (illustrative, example prices)Forgottenclusterrunning the whole time, almost all of it idle: about $139Job clusters,auto-stopthree nights of real work, then the machines stop: about $5FriSatSunMon09:00
Compute is most of the bill, and idle compute costs the same as busy compute. Example prices from the weekend-bill story in our performance stories.

Our performance stories include one worked example, with example prices rather than a real bill. An all-purpose cluster of 4 nodes is started on Friday evening to debug a job and never stopped. By Monday morning it has run for 63 hours and cost about $139, for a weekend with no new data. The same three nights of real work on job clusters, with auto-termination on the interactive cluster, comes to about $5. Your rates will differ. The shape of the saving is the point.

The compute leaks

1. Interactive compute running scheduled jobs

All-purpose compute is the cluster you attach a notebook to and explore with. Job compute is started by a job and stopped when the job ends, and it is billed at a lower rate. Scheduled jobs that point at an all-purpose cluster pay the higher rate, and they keep that cluster alive between runs.

2. Clusters too big, or autoscaling that never shrinks

A cluster sized for last year's biggest backfill now runs every small daily job. Or autoscaling has a minimum so high that the cluster can never get smaller, which is what happened in the weekend example: the minimum was 4, so it stayed at 4.

3. No auto-termination

An interactive cluster with auto-termination off will run until someone remembers it.

4. SQL warehouses left on, or sized too big

SQL warehouses are the compute behind dashboards and SQL queries. A large warehouse that never stops, serving a dashboard opened twice a day, is a quiet leak.

The data and code leaks

These don't show up as an idle cluster. They show up as a job that takes an hour when it should take five minutes, and you pay for every one of those minutes.

5. Small files

A streaming job or a frequent small batch writes thousands of tiny files. Every read must list them, open them and read each file's footer, which costs more than reading the data. The rule we use: flag any table whose average file is under about 32 MB.

same data, same answer; the table just stops being 3,000 pieces3,000 files of about 7 KBquery: 1.4 sOPTIMIZEcompacts themin one commit1 filequery: 235 msthe rule we use:flag any table whoseaverage file is underabout 32 MB
Small files, before and after OPTIMIZE. Timings are from our recorded laptop run, so compare before with after, not with a cluster.

6. Skew: one task does most of the work

Spark splits work into tasks that run side by side. If 40% of orders belong to one customer, say a "guest checkout" account, every one of those rows lands in one task. The other tasks finish and wait. The cluster is paid for while it waits.

the join stage, 16 tasks: everyone waits for the long onetask 1task 16median: 45 msslowest: 246 ms5.5 times the median.The rule we use:skew when the slowesttask is over 5 timesthe medianthe other 15 finish fast,then sit and wait
Skew in our recorded join: one customer holds 40% of the orders, so one task does far more work than the rest.

7. Too many or too few shuffle partitions

A shuffle is when Spark moves rows between machines, for a join or a group-by. The number of pieces it cuts the data into matters both ways. Too few, and each task gets more than fits in memory and writes to disk, called spill. Too many, and Spark spends its time scheduling thousands of tiny tasks.

8. Python UDFs where a built-in function would do

A Python UDF (user-defined function) sends every row out of Spark's engine to a Python process and back. Photon, Databricks' faster engine, cannot speed it up either.

9. collect() and toPandas() on big data

Both pull every row onto the driver, the one machine that coordinates the cluster. On a big table the driver runs out of memory, the job fails, and someone "fixes" it with a bigger driver.

The leak that hides all the others

10. No cost tags

Without tags, the bill is one number. Nobody can say which team, project or job spent it, so nobody feels responsible for it.

A ten-minute weekly check

You don't need a cost tool to start. You need one query and one habit.

every Monday, ten minutes1Billing tablessystem.billing.usagelast 7 days2Group by tagteam, project, job;flag what's untagged3A dashboardtop five spenders,a budget alertone cup of chaitags make step 2 possible, so they come first
A ten-minute weekly cost check. No cost tool needed to start.
  1. Query the billing system tables (system.billing.usage, joined to system.billing.list_prices) for the last 7 days, grouped by tag and by compute type.
  2. Look for the top five jobs and clusters by spend, and anything untagged.
  3. Put that query on a dashboard, add a budget alert, and look at it every Monday with a cup of chai.

Ten questions to check your own workspace

  1. Which scheduled jobs run on all-purpose compute?
  2. What is the smallest size each cluster can autoscale down to?
  3. Does any all-purpose cluster have auto-termination switched off?
  4. How many queries did your biggest SQL warehouse run last week, and how long was it on?
  5. Which of your largest tables have an average file size under 32 MB?
  6. In your slowest job, how does the longest task compare with the median?
  7. Is the shuffle partition count set by hand anywhere, and does any stage spill?
  8. How many Python UDFs could be built-in functions?
  9. Where does code call collect() or toPandas() on a whole table?
  10. What share of last month's bill has no team tag?

Fix the holes in the bucket before you buy a bigger tap.