Home › Learn › Data systems › Data platform › Cost & FinOps

Data platform · concern X10 · Both

Cost & FinOps: end to end and per hop

Every dollar has an owner and a budget, end to end and per hop, and the hidden costs are counted before they surprise anyone.

What a data platform costs, how to read the bill end to end and per hop, the drivers at each layer, the hidden costs, and how streaming and real-time serving change the picture.

For leaders

In plain words

Like the electricity bill for a house

A house's electricity bill has the appliances you use, the standing charge you pay anyway, and the things nobody notices: the TV on standby, the heater left on in an empty room. You can read it two ways. For the whole house, "is this worth it?". Room by room, "what's using so much?". A separate meter per room makes both possible.

A data platform's bill is the same. The appliances are the hops: ingestion, merges, gold refreshes, warehouses. The standing charges are storage and always-on services. The standby lights are the hidden costs. Tags are the meters.

DatabricksDBUs: jobs, pipelines,warehouses, notebooksAWS underneathEC2 (classic only),S3, networkHiddennever stops, retries,dev/test, maintenanceserverless folds the machines into the DBU pricestorage and network are billed by AWSoften a third of the billend to end: one budget per data product · per hop: a few drivers each · tags make both visible
The three parts of the bill, and the two ways to look at it.

Open the Databricks cost calculator

Two views

End to end, per hop, and per team

ViewThe question it answersWho owns itHow it is measured
End to endWhat does this data product cost, all in, and is it worth it?The data product ownerUsage tagged by product, summed across every hop, plus its share of shared costs
Per hopWhich layer is expensive, and why?The platform teamUsage tagged by hop (job, pipeline or warehouse), with the hop's drivers next to it
Per teamWho spends on notebooks, monitoring and ad-hoc queries?Team leadsUsage of notebooks and warehouses tagged by team

Per hop

What drives the cost at each hop

HopWhat drives the costThe main lever
H1 · Sources to bronzeData volume and file format; connectors and CDC; runs per day; landing and bronze retentionIncremental ingestion (Auto Loader); keep landing short; bronze history only as long as you need to replay
H2 · Bronze to silverMerges on big tables: how many files each merge rewrites; history-keeping (SCD2); runs per dayLiquid clustering and deletion vectors; fewer, larger runs; history only where it matters
H3 · Silver to goldFull rebuilds versus incremental refresh; number of materialized viewsIncremental materialized views; drop gold tables nobody reads
H4 · Gold to semanticWhether dashboards scan big fact tables or small aggregatesPre-computed aggregates behind metric views
H5 · Semantic to experiencesWarehouse size and hours on; refresh schedules; number of users; Genie and agentsServerless warehouses with short auto-stop; refresh schedules that match reading habits
Streaming laneAlways-on pipelines; how many tables share a pipelineTrigger every few minutes where seconds aren't needed; pack tables into fewer pipelines

Hidden costs

The costs no plan includes

CostWhy it hidesHow to see itFix
A warehouse that never stopsEach refresh is cheap; together they keep it on all nightWarehouse hours outside business hours, in system.billing.usageBusiness-hours schedules; short auto-stop
Dev and testIt is 'not production', so nobody watches itUsage tagged by environmentSmaller compute, auto-stop, sampled data
Failed runs and retriesThe job finally succeeded, so nobody looks backRun history: compute spent on failed attemptsFix the top failing jobs; cap retries
Table maintenanceOPTIMIZE and VACUUM run in the backgroundPredictive optimization and maintenance jobs in billingLet predictive optimization decide; skip tables nobody reads
Old file versionsEvery merge leaves old files until VACUUMStorage growing faster than the dataShort time-travel retention where replay isn't needed
Monitoring by teamsEach team builds its own cost and health dashboardsSmall warehouses running all nightOne shared monitoring warehouse; hourly refresh in business hours
Idle notebooksEngineers stop typing; clusters keep billingAll-purpose clusters idle before auto-terminationServerless notebooks, or short auto-termination by policy
Cluster start-up and NATClassic compute pays EC2 while starting; S3 traffic without a gateway endpoint pays NAT per GBEC2 and NAT lines on the AWS billServerless, or an S3 gateway endpoint and pools
BackfillsPlanned as a one-off, repeated twice a yearUsage spikes after releasesPlan backfills as jobs with a cost estimate

Evidence

From estimate to evidence

Estimate first with the calculator. Once the platform runs, the truth is in Databricks' own billing tables: system.billing.usage (what ran, how many DBUs, with its tags) and system.billing.list_prices (the price of each product). Join them, group by the product, hop, team and environment tags, and you have both views. Add the AWS bill for EC2, S3 and network, split by the same tags.

The question list

Every question, why we ask, and what breaks if nobody does

#Question to chaseWhy we askWhat breaks in production if nobody asksDo this (not this)
A · Ownership and budgetswho pays, and against what
A1Does every data product have a named owner and a monthly budget?Cost without an owner only grows.The bill triples in a quarter and every team says it's someone else's.A budget per data product, with alerts at 80% and 100%. Not: one platform budget nobody owns.
A2Is every job, pipeline and warehouse tagged with product, hop, team and environment?Without tags, the bill can't be split end to end or per hop.Half the bill shows up as 'untagged' and can't be explained.Compute policies that require the tags; reject untagged compute. Not: tagging by convention.
A3Do we show costs to teams first, and charge them later?Charging before people trust the numbers starts arguments, not savings.Teams dispute the chargeback instead of fixing the cost.Show back for a quarter, then charge back. Not: charging on day one.
B · Per hopwhere each layer's money goes
B1Which tables cost the most to merge, and how is their file layout?Merges on big, badly laid-out tables are usually the largest pipeline cost.One 50 TB table rewrites a fifth of its files every run and nobody knows.Cost per table from billing and job metrics; liquid clustering on the top ten. Not: tuning small tables first.
B2Which gold tables rebuild fully, and does anyone read them?Full rebuilds re-read all of silver, every run.Gold refresh costs more than everything upstream, for tables last queried months ago.Incremental refresh; retire unread tables using lineage and query history. Not: keeping every table forever.
B3How long are warehouses on, and how many hours is anyone actually querying?Warehouses are billed per hour on, not per query.A warehouse runs 730 hours a month for 200 hours of queries.Compare uptime with query hours; fix the schedules keeping it awake. Not: sizing up when it feels slow.
C · Hidden coststhe costs no plan includes
C1What do dev and test cost compared with production?Lower environments are rarely watched.Test costs as much as production because a full-size job runs on a schedule there.Tag by environment and set a target share (20-40%). Not: assuming non-prod is small.
C2What runs at night and at weekends, and why?Off-hours usage is usually refreshes, monitoring and forgotten jobs.Monitoring dashboards keep a warehouse awake every night.A weekly look at off-hours usage. Not: discovering it in the annual review.
C3How much compute goes on failed runs and retries?A job that succeeds on the third try costs three times.A flaky job doubles its cost for months.Track compute on failed attempts; fix the top offenders. Not: unlimited retries.
D · Streaming and real timealways-on, on purpose
D1Which tables need seconds, which need minutes, and which are fine daily?Always-on compute costs the same whether anyone looks or not.Twenty tables stream continuously for a dashboard read once a morning.Decide speed per table with its user; trigger every few minutes where seconds aren't needed. Not: streaming by default.
D2What answers real-time questions at night: a warehouse, a database, or an API?Real-time serving means something is always on.A large warehouse kept up 24x7 to answer a handful of API calls.Size the always-on part for the night-time load, and serve point lookups from a database or cache. Not: the daytime warehouse left on.
E · Measuringfrom estimate to evidence
E1Do we read cost from Databricks' billing tables, joined with list prices and tags?Estimates are for planning; the billing tables are the truth.The estimate becomes the budget and nobody checks it against reality.A cost dashboard on system.billing.usage and system.billing.list_prices, per product and per hop. Not: the monthly invoice alone.
E2Do we compare the estimate with the real bill after the first month?The calculator's work rates are calibrated defaults, not your data.A design decision rests on a number that was off by 3x.Replace the calculator's work rates with measured ones and re-run the estimate. Not: trusting any calculator, ours included.

Design decisions

The decisions to take

Serverless or classic compute for pipelines?

OptionVerdict
Serverless by defaultPursue · Pursue. No idle clusters, start-up time, NAT or tuning; costs follow the work.
Classic with spot workers for long, steady, large jobsPursue when · Pursue only where the calculator and billing show a clear saving and the team can tune clusters.
Classic everywhere because the DBU price is lowerDrop · Drop. The EC2, start-up, idle and network costs come on top.

How are costs split?

OptionVerdict
Tags required by compute policy; show back, then charge backPursue · Pursue. Both views (end to end and per hop) come from the same tags.
Split the bill by headcountDrop · Drop. It hides who drives the cost.

How do streaming tables run?

OptionVerdict
Triggered every few minutes, many tables per pipelinePursue · Pursue for most 'near real time' needs.
ContinuousPursue when · Pursue only where a decision needs seconds (fraud, stock, operations).

How are dashboards refreshed?

OptionVerdict
On a schedule that matches when people read themPursue · Pursue. Business hours, and hourly at most for most dashboards.
Every 15 minutes, around the clockDrop · Drop unless someone reads it at night. The warehouse never stops.

FAQ

Questions people ask

What makes up the cost of a Databricks data platform?

Three parts: Databricks DBUs (for jobs, pipelines, SQL warehouses and notebooks), the cloud resources underneath (machines for classic compute, storage, network), and hidden costs such as warehouses that never stop, dev and test, retries, maintenance and monitoring.

Is serverless more expensive than classic compute on Databricks?

The price per DBU is higher, but it includes the machines, and you don't pay for start-up, idle clusters or NAT. For bursty and scheduled work serverless is often cheaper overall; for long, steady jobs on spot machines classic can be cheaper. The calculator shows both for your settings.

Should Databricks cost be tracked per layer or end to end?

Both. End to end per data product, so cost sits next to value; per hop, so the platform team knows which layer to fix. Both views come from the same tags on jobs, pipelines and warehouses.

What are the hidden costs of a Databricks platform?

Warehouses kept awake by scheduled refreshes, dev and test environments, failed runs and retries, table maintenance, old file versions in storage, teams' monitoring dashboards, idle notebook clusters, and on classic compute, cluster start-up time and NAT gateway traffic.

Next

Where to go next

Try it: the Databricks data platform cost calculator, per hop and end to end, with streaming and hidden costs.

Read more: Your Databricks bill has a leak and Your cluster scales up and never comes back down (blog).

Back up: The data platform, whole, with all 24 concerns.