Home › Data engineering › Fundamentals › Cost
Data engineering fundamentals · free guide
Cost fundamentals: what you pay for in a data platform
A data bill is not a mystery either. You pay for four things: compute time, stored bytes, bytes that move, and the people and services around them. Most waste is one of a short list of patterns. This page organises them as a mind map, with the matching CostSherlock rules, so you can find the leak by category.
Start here
The data physics
Five plain facts that explain almost everything below. Read these and the rest follows.
The mind map
The whole topic on one screen
Each card is a branch. Walk them in order when you are stuck: the question tells you what the branch is about, and the first check tells you where to look.
1. Compute that is on
Are machines running when no work is?
Idle clusters, no auto-termination, over-large warehouses, long auto-stop
First check: Cluster hours vs busy hours
2. Compute that is too big
Is it sized for the job?
Over-sized driver or nodes, autoscale max, memory-optimised without need
First check: CPU and memory use against size
3. Compute that is the wrong kind
Right tool for the work?
Interactive for scheduled jobs, classic vs serverless, no spot, old generations
First check: Job type against compute type
4. Queries that waste
Do queries read or compute more than they need?
Full scans, SELECT *, cartesian joins, spill, recompute
First check: Bytes scanned vs bytes used
5. Storage that grows
Why is stored data growing?
Small files, old versions, log growth, duplicate copies, unused tables
First check: Storage by table and age
6. Refresh that is too frequent
Does anyone need it this fresh?
Over-frequent schedules, dashboards, subscriptions, continuous pipelines for batch
First check: Refresh rate vs read rate
7. Data that moves
What crosses a boundary?
Cross-region, cross-cloud, egress, BI tools pulling extracts
First check: Network and egress lines
8. Spend nobody explains
Who owns this?
Untagged resources, orphaned objects, spikes, new unattributed workloads
First check: Share of cost without an owner
Where the money goes
The five lines on every data bill
Whatever the vendor, the invoice breaks into the same kinds of line. Knowing the line tells you which lever to pull.
The five lines of a bill. The split varies by team; use the shape, not the numbers.
| Cost line | What you are charged for | Main driver | Biggest lever | Verdict |
|---|---|---|---|---|
| Compute | Machine time (VMs, units of processing, warehouses) | Hours on, size, count | Switch off when idle; right-size; make jobs faster | Usually the biggest line; start here |
| Storage | Bytes kept, including old versions and logs | Volume and retention | Delete unused data, compact files, limit history | Usually small; surprising when ignored |
| Network / egress | Bytes leaving a region or cloud | Cross-region reads, extracts | Co-locate compute and data | Small until it is not |
| Platform services | Orchestration, catalogs, AI function calls, vector search, serving endpoints | Calls, tokens, uptime | Cap, cache, scale to zero | Growing fast; watch it |
| People and time | Engineers fixing, re-running, reconciling | Rework and firefighting | Fewer incidents, earlier checks | Often larger than the cloud bill |
Compute that is on
Idle is the most common waste
A cluster that waits for work still bills. Interactive clusters left on overnight, warehouses that auto-stop after an hour, jobs that keep a cluster warm between runs: all pay for nothing.
Idle compute: the same bill for busy and idle hours. Illustrative example.
| Pattern | Why it costs | Fix | Verdict |
|---|---|---|---|
| No auto-termination | Cluster runs until someone remembers | Set a short idle timeout | Always fix |
| Long auto-stop on warehouses | Pays for the whole idle window each time | Shorten to minutes | Pursue |
| Interactive cluster running scheduled jobs | Interactive rates for non-interactive work, and it stays on | Move to job compute | Pursue |
| Always-warm pools nobody uses | Pay to keep machines ready | Reduce or remove | Pursue |
| Continuous pipeline on batch data | Streaming cost for data that arrives hourly | Triggered, scheduled runs | Pursue |
| Orphaned clusters of departed users | Nobody owns them | Policy: owner required, auto-clean | Always fix |
Compute that is too big or wrong
Right-size, then right-kind
Bigger machines are not faster unless the job can use them. Look at what the machines actually did before paying for more.
| Pattern | Signal | Fix | Verdict |
|---|---|---|---|
| Over-sized nodes or driver | CPU and memory mostly idle | Smaller instances, smaller driver | Pursue |
| Autoscale maximum far above need | Scales out and sits idle | Lower the cap | Pursue |
| Memory-optimised without memory pressure | Pays a premium for unused memory | General purpose | Pursue |
| Single node running a distributed workload | Slow, and the long run costs more | Right-size workers | Pursue |
| No spot instances for fault-tolerant work | Pays full price | Spot workers, on-demand driver | Pursue for retry-safe jobs |
| Spot used with retry storms | Interruptions rerun work | Mix on-demand, checkpoint | Pursue |
| Old runtime or instance generation | Slower per dollar | Upgrade | Pursue |
| Photon-style acceleration without benefit | Higher unit price, no speed gain | Measure; turn off where it does not pay | Measure first |
Queries that waste
Performance waste is cost waste
Every wasted byte scanned or shuffled is paid for twice: in compute time now, and in a slower finish. The performance page covers the mechanics; here is how it appears on the bill.
| Pattern | Why it costs | Fix | Verdict |
|---|---|---|---|
| Recurring full table scans | Same bytes read again and again | Layout for the common filter; precompute | Pursue first |
| SELECT * on wide tables | Reads columns nobody needs | Name the columns | Pursue |
| Loops running a query or action per pass | Fixed start-up cost multiplied | One set-based query | Always fix |
| Dashboards hitting raw (bronze) tables | Heavy query on every open | Point at gold | Pursue |
| Spill and skew | Machines held for the straggler | Fix partitions and keys | Pursue |
| Cartesian joins at scale | Explosive work | Add the join condition | Always fix |
Storage that grows
Small files, old versions and forgotten tables
Storage is cheap per byte, but table maintenance decides whether storage stays small and fast. Neglected tables get slower and larger together.
| Pattern | Why it costs | Fix | Verdict |
|---|---|---|---|
| Small files | More requests, more tasks, slower reads | Compact regularly | Pursue |
| Vacuum / clean-up overdue | Deleted files still stored and billed | Schedule clean-up | Pursue |
| Over-long history retention | Old versions kept for nothing | Retain what you actually time-travel | Pursue |
| Change feed on with no consumer | Extra data written for nobody | Turn off | Pursue |
| Maintenance run too often | Compaction thrash rewrites files again and again | Run on need, not on a timer | Pursue |
| Tables and views with no readers | Built and refreshed for nobody | Retire them | Pursue |
Refresh that is too frequent
Freshness is a product decision
Every refresh is a cost. The question is not 'can we refresh every minute?' but 'who reads it that often?'
Frequency multiplies everything. Same job, same $2: monthly is $24 a year, hourly is $17,520.
| Pattern | Why it costs | Fix | Verdict |
|---|---|---|---|
| Cron more frequent than data changes | Runs on unchanged data | Match to arrival | Pursue |
| Dashboard auto-refresh too fast | Queries run whether anyone watches | Slower refresh; cache | Pursue |
| Subscriptions emailed too often | Same query each send | Reduce frequency | Pursue |
| Dashboards nobody opens | Refresh with no reader | Retire or pause | Always fix |
| Query per viewer fan-out | N viewers, N queries | Shared cached result | Pursue |
| BI tool direct query on heavy data | Each click is a warehouse query | Import or aggregate in gold | Pursue |
Data that moves
Egress and cross-boundary reads
Moving data between regions or clouds costs per byte and adds latency. Keep compute near data.
| Pattern | Why it costs | Fix | Verdict |
|---|---|---|---|
| Compute and storage in different regions | Egress on every read | Co-locate | Pursue |
| Data in several clouds | Egress and duplicate copies | Pick a home; share, do not copy | Pursue |
| Large extracts to external tools | Pays egress and loses control | Query in place | Pursue |
| Chatty orchestration | Many tiny invocations from outside | Batch calls | Pursue |
Spend nobody explains
Anomalies, tags and ownership
You cannot manage a cost you cannot attribute. Tags and owners turn a bill into a conversation.
| Pattern | Signal | Fix | Verdict |
|---|---|---|---|
| No cost tags | Spend cannot be assigned to a team | Required tags by policy | Always fix |
| New workload with no owner | Sudden unattributed line | Assign an owner on creation | Pursue |
| Day-over-day spike | One day far above normal | Alert and trace the job | Pursue |
| Sustained shift upward | A new baseline after a change | Review the change | Pursue |
| One user dominates spend | Runaway personal compute | Quota and review | Pursue |
| Seasonality ignored | Weekly cycle triggers false alarms | Compare to the same weekday | Tune alerts |
In practice
CostSherlock rules for this topic
CostSherlock is our open set of cost and performance rules. Each rule below has a plain meaning here; the rule id is its name in CostSherlock. They are grouped by the same branches as the mind map, so a finding tells you which branch to open.
Compute that is on
| CostSherlock rule | What it catches, in plain words |
|---|---|
cluster_no_autotermination | No idle shutdown |
uc_cluster_idle_no_workload | Cluster up with no work |
uc_warehouse_auto_stop_too_long | Warehouse waits too long to stop |
uc_cluster_long_lived_interactive | Interactive cluster kept for days |
uc_interactive_cluster_for_scheduled_job | Scheduled job on an interactive cluster |
uc_warehouse_dormant_with_provisioned_clusters | Warehouse idle but still provisioned |
dlt_continuous_for_batch_workload | Continuous pipeline for batch data |
Compute that is too big or wrong
| CostSherlock rule | What it catches, in plain words |
|---|---|
uc_cluster_oversized_for_workload | Cluster too large for the work |
oversized_autoscale_max | Autoscale ceiling too high |
uc_cluster_oversized_driver | Driver bigger than needed |
uc_cluster_memory_optimized_unused | Memory-optimised, memory unused |
uc_cluster_no_spot_workers | No spot workers where safe |
uc_spot_cluster_retry_storm | Spot interruptions rerunning work |
uc_cluster_old_instance_generation | Old instance generation |
uc_photon_overhead_negative_roi | Acceleration costs more than it saves |
uc_oversized_warehouse_low_utilization | Big warehouse, low use |
Queries that waste
| CostSherlock rule | What it catches, in plain words |
|---|---|
uc_query_full_scan_recurring | Recurring full scans |
uc_select_star_at_scale | SELECT * on large data |
uc_query_cartesian_join_at_scale | Cartesian join at scale |
uc_query_spill_to_disk_recurring | Recurring spill |
uc_query_recompute_no_caching | Recompute without caching |
code_spark_action_inside_loop | Action inside a loop |
collect_in_for_loop | collect() in a loop |
Storage that grows
| CostSherlock rule | What it catches, in plain words |
|---|---|
too_many_files | Too many small files |
uc_vacuum_overdue | Clean-up overdue |
delta_vacuum_overdue | Vacuum overdue |
delta_log_growth_without_checkpoint | Transaction log growing unchecked |
uc_cdf_enabled_no_consumer | Change feed with no reader |
uc_optimize_thrash_table | Compaction thrash |
uc_mv_no_readers | Materialised view nobody reads |
uc_delta_time_travel_unused | History kept but never used |
Refresh that is too frequent
| CostSherlock rule | What it catches, in plain words |
|---|---|
cron_too_frequent | Schedule too frequent |
uc_dashboard_auto_refresh_too_frequent | Dashboard refreshes too often |
uc_dashboard_subscription_over_frequent | Subscriptions too frequent |
uc_dashboard_nobody_opens | Dashboard nobody opens |
uc_dashboard_query_per_viewer_fan_out | A query per viewer |
uc_external_bi_tool_direct_query_heavy | BI tool querying directly and heavily |
adf_trigger_too_frequent | Pipeline trigger too frequent |
Data that moves
| CostSherlock rule | What it catches, in plain words |
|---|---|
azure_excessive_egress | Excessive egress |
uc_external_locations_multi_cloud | Locations across clouds |
uc_external_locations_region_mixed | Locations across regions |
adf_data_movement_dominates_pipeline_cost | Data movement dominates the pipeline |
Spend nobody explains
| CostSherlock rule | What it catches, in plain words |
|---|---|
uc_cluster_no_cost_tags | No cost tags |
cost_anomaly_day_spike | Daily spike |
cost_anomaly_sustained_shift | Sustained shift |
cost_anomaly_new_unattributed_workload | New unattributed workload |
cost_anomaly_dow_seasonality_aware | Weekly pattern considered |
uc_workspace_runaway_cost_by_user | One user runs away with spend |
azure_orphaned_managed_disk | Orphaned disk |
dbx_orphaned_vm_after_cluster_terminated | VM left after cluster ended |
FAQ
Questions people ask
Where do most data platform bills leak?
Idle compute, over-sized compute, and the same expensive query or refresh repeated for no reader. Storage and network are usually smaller lines, but neglected tables slow everything and quietly grow.
Is a faster job always cheaper?
Usually, because machines are held for less time. Paying for a much larger cluster to finish sooner can cost more if the job does not scale with it. Measure cost per run, not just run time.
How often should a pipeline run?
As often as someone acts on the result, and no more. Ask who reads it and what they do differently with a fresher answer.
Does serverless always save money?
No. It removes start-up waits and idle waste for spiky work, and can cost more for steady heavy work. Compare on your own pattern.
What is the first thing to do about cost?
Tag everything with an owner, then look at the top ten items by spend. You cannot fix what nobody owns.
Are the sample figures real?
They are illustrative arithmetic to show how frequency multiplies a cost, not quotes. Prices differ by vendor and contract.
Related
Where it connects
Performance fundamentals · Processing fundamentals · Security fundamentals · Storage fundamentals · Database families · Where data engineering time goes · Who maintains your tables · All data engineering
The numbers and thresholds on this page are rules of thumb and orders of magnitude to build intuition, not benchmarks. Check your platform's current documentation and measure your own workload.