Home › Learn › Data systems › Data platform › Cost & FinOps
Data platform · concern X10 · Both
Cost & FinOps: end to end and per hop
Every dollar has an owner and a budget, end to end and per hop, and the hidden costs are counted before they surprise anyone.
What a data platform costs, how to read the bill end to end and per hop, the drivers at each layer, the hidden costs, and how streaming and real-time serving change the picture.
For leaders
- A Databricks bill has three parts: Databricks DBUs, the AWS resources underneath (machines for classic compute, storage, network), and hidden costs nobody planned.
- Look at it two ways. End to end: one budget per data product, so value and cost sit side by side. Per hop: a few cost drivers per layer, so the platform team knows where to act.
- Hidden costs are often a third of the bill: warehouses that never stop, dev and test, retries, maintenance, monitoring dashboards, idle notebooks and file history.
- Streaming and real-time serving are always-on costs. Choose them per table, for a decision that needs seconds or minutes.
- Estimate before building with the calculator; once running, measure with Databricks' own billing tables and tags.
In plain words
Like the electricity bill for a house
A house's electricity bill has the appliances you use, the standing charge you pay anyway, and the things nobody notices: the TV on standby, the heater left on in an empty room. You can read it two ways. For the whole house, "is this worth it?". Room by room, "what's using so much?". A separate meter per room makes both possible.
A data platform's bill is the same. The appliances are the hops: ingestion, merges, gold refreshes, warehouses. The standing charges are storage and always-on services. The standby lights are the hidden costs. Tags are the meters.
Two views
End to end, per hop, and per team
| View | The question it answers | Who owns it | How it is measured |
|---|---|---|---|
| End to end | What does this data product cost, all in, and is it worth it? | The data product owner | Usage tagged by product, summed across every hop, plus its share of shared costs |
| Per hop | Which layer is expensive, and why? | The platform team | Usage tagged by hop (job, pipeline or warehouse), with the hop's drivers next to it |
| Per team | Who spends on notebooks, monitoring and ad-hoc queries? | Team leads | Usage of notebooks and warehouses tagged by team |
Per hop
What drives the cost at each hop
| Hop | What drives the cost | The main lever |
|---|---|---|
| H1 · Sources to bronze | Data volume and file format; connectors and CDC; runs per day; landing and bronze retention | Incremental ingestion (Auto Loader); keep landing short; bronze history only as long as you need to replay |
| H2 · Bronze to silver | Merges on big tables: how many files each merge rewrites; history-keeping (SCD2); runs per day | Liquid clustering and deletion vectors; fewer, larger runs; history only where it matters |
| H3 · Silver to gold | Full rebuilds versus incremental refresh; number of materialized views | Incremental materialized views; drop gold tables nobody reads |
| H4 · Gold to semantic | Whether dashboards scan big fact tables or small aggregates | Pre-computed aggregates behind metric views |
| H5 · Semantic to experiences | Warehouse size and hours on; refresh schedules; number of users; Genie and agents | Serverless warehouses with short auto-stop; refresh schedules that match reading habits |
| Streaming lane | Always-on pipelines; how many tables share a pipeline | Trigger every few minutes where seconds aren't needed; pack tables into fewer pipelines |
Evidence
From estimate to evidence
Estimate first with the calculator. Once the platform runs, the truth is in Databricks' own billing tables: system.billing.usage (what ran, how many DBUs, with its tags) and system.billing.list_prices (the price of each product). Join them, group by the product, hop, team and environment tags, and you have both views. Add the AWS bill for EC2, S3 and network, split by the same tags.
The question list
Every question, why we ask, and what breaks if nobody does
| # | Question to chase | Why we ask | What breaks in production if nobody asks | Do this (not this) |
|---|---|---|---|---|
| A · Ownership and budgetswho pays, and against what | ||||
| A1 | Does every data product have a named owner and a monthly budget? | Cost without an owner only grows. | The bill triples in a quarter and every team says it's someone else's. | A budget per data product, with alerts at 80% and 100%. Not: one platform budget nobody owns. |
| A2 | Is every job, pipeline and warehouse tagged with product, hop, team and environment? | Without tags, the bill can't be split end to end or per hop. | Half the bill shows up as 'untagged' and can't be explained. | Compute policies that require the tags; reject untagged compute. Not: tagging by convention. |
| A3 | Do we show costs to teams first, and charge them later? | Charging before people trust the numbers starts arguments, not savings. | Teams dispute the chargeback instead of fixing the cost. | Show back for a quarter, then charge back. Not: charging on day one. |
| B · Per hopwhere each layer's money goes | ||||
| B1 | Which tables cost the most to merge, and how is their file layout? | Merges on big, badly laid-out tables are usually the largest pipeline cost. | One 50 TB table rewrites a fifth of its files every run and nobody knows. | Cost per table from billing and job metrics; liquid clustering on the top ten. Not: tuning small tables first. |
| B2 | Which gold tables rebuild fully, and does anyone read them? | Full rebuilds re-read all of silver, every run. | Gold refresh costs more than everything upstream, for tables last queried months ago. | Incremental refresh; retire unread tables using lineage and query history. Not: keeping every table forever. |
| B3 | How long are warehouses on, and how many hours is anyone actually querying? | Warehouses are billed per hour on, not per query. | A warehouse runs 730 hours a month for 200 hours of queries. | Compare uptime with query hours; fix the schedules keeping it awake. Not: sizing up when it feels slow. |
| C · Hidden coststhe costs no plan includes | ||||
| C1 | What do dev and test cost compared with production? | Lower environments are rarely watched. | Test costs as much as production because a full-size job runs on a schedule there. | Tag by environment and set a target share (20-40%). Not: assuming non-prod is small. |
| C2 | What runs at night and at weekends, and why? | Off-hours usage is usually refreshes, monitoring and forgotten jobs. | Monitoring dashboards keep a warehouse awake every night. | A weekly look at off-hours usage. Not: discovering it in the annual review. |
| C3 | How much compute goes on failed runs and retries? | A job that succeeds on the third try costs three times. | A flaky job doubles its cost for months. | Track compute on failed attempts; fix the top offenders. Not: unlimited retries. |
| D · Streaming and real timealways-on, on purpose | ||||
| D1 | Which tables need seconds, which need minutes, and which are fine daily? | Always-on compute costs the same whether anyone looks or not. | Twenty tables stream continuously for a dashboard read once a morning. | Decide speed per table with its user; trigger every few minutes where seconds aren't needed. Not: streaming by default. |
| D2 | What answers real-time questions at night: a warehouse, a database, or an API? | Real-time serving means something is always on. | A large warehouse kept up 24x7 to answer a handful of API calls. | Size the always-on part for the night-time load, and serve point lookups from a database or cache. Not: the daytime warehouse left on. |
| E · Measuringfrom estimate to evidence | ||||
| E1 | Do we read cost from Databricks' billing tables, joined with list prices and tags? | Estimates are for planning; the billing tables are the truth. | The estimate becomes the budget and nobody checks it against reality. | A cost dashboard on system.billing.usage and system.billing.list_prices, per product and per hop. Not: the monthly invoice alone. |
| E2 | Do we compare the estimate with the real bill after the first month? | The calculator's work rates are calibrated defaults, not your data. | A design decision rests on a number that was off by 3x. | Replace the calculator's work rates with measured ones and re-run the estimate. Not: trusting any calculator, ours included. |
Design decisions
The decisions to take
Serverless or classic compute for pipelines?
| Option | Verdict |
|---|---|
| Serverless by default | Pursue · Pursue. No idle clusters, start-up time, NAT or tuning; costs follow the work. |
| Classic with spot workers for long, steady, large jobs | Pursue when · Pursue only where the calculator and billing show a clear saving and the team can tune clusters. |
| Classic everywhere because the DBU price is lower | Drop · Drop. The EC2, start-up, idle and network costs come on top. |
How are costs split?
| Option | Verdict |
|---|---|
| Tags required by compute policy; show back, then charge back | Pursue · Pursue. Both views (end to end and per hop) come from the same tags. |
| Split the bill by headcount | Drop · Drop. It hides who drives the cost. |
How do streaming tables run?
| Option | Verdict |
|---|---|
| Triggered every few minutes, many tables per pipeline | Pursue · Pursue for most 'near real time' needs. |
| Continuous | Pursue when · Pursue only where a decision needs seconds (fraud, stock, operations). |
How are dashboards refreshed?
| Option | Verdict |
|---|---|
| On a schedule that matches when people read them | Pursue · Pursue. Business hours, and hourly at most for most dashboards. |
| Every 15 minutes, around the clock | Drop · Drop unless someone reads it at night. The warehouse never stops. |
FAQ
Questions people ask
What makes up the cost of a Databricks data platform?
Three parts: Databricks DBUs (for jobs, pipelines, SQL warehouses and notebooks), the cloud resources underneath (machines for classic compute, storage, network), and hidden costs such as warehouses that never stop, dev and test, retries, maintenance and monitoring.
Is serverless more expensive than classic compute on Databricks?
The price per DBU is higher, but it includes the machines, and you don't pay for start-up, idle clusters or NAT. For bursty and scheduled work serverless is often cheaper overall; for long, steady jobs on spot machines classic can be cheaper. The calculator shows both for your settings.
Should Databricks cost be tracked per layer or end to end?
Both. End to end per data product, so cost sits next to value; per hop, so the platform team knows which layer to fix. Both views come from the same tags on jobs, pipelines and warehouses.
What are the hidden costs of a Databricks platform?
Warehouses kept awake by scheduled refreshes, dev and test environments, failed runs and retries, table maintenance, old file versions in storage, teams' monitoring dashboards, idle notebook clusters, and on classic compute, cluster start-up time and NAT gateway traffic.
Next
Where to go next
Try it: the Databricks data platform cost calculator, per hop and end to end, with streaming and hidden costs.
Read more: Your Databricks bill has a leak and Your cluster scales up and never comes back down (blog).
Back up: The data platform, whole, with all 24 concerns.