Home › Data engineering › Fundamentals › Cost

Data engineering fundamentals · free guide

Cost fundamentals: what you pay for in a data platform

A data bill is not a mystery either. You pay for four things: compute time, stored bytes, bytes that move, and the people and services around them. Most waste is one of a short list of patterns. This page organises them as a mind map, with the matching CostSherlock rules, so you can find the leak by category.

Start here

The data physics

Five plain facts that explain almost everything below. Read these and the rest follows.

You pay for time, not work.Compute is billed by the second or hour that it is on. An idle cluster costs the same as a busy one.
Performance and cost are the same fight.A faster job usually costs less because it holds the machine for less time. The exception is paying for speed you did not need.
Storage is cheap, scanning it is not.Keeping a byte costs little. Reading it again and again, or moving it between regions, is where money goes.
Frequency multiplies everything.A job that costs 2 dollars is 730 dollars a year if it runs daily and 17,500 if it runs hourly. Ask how often before asking how fast.
What nobody owns, nobody stops.Orphaned clusters, unused dashboards and forgotten schedules run until someone is made responsible.

The mind map

The whole topic on one screen

Each card is a branch. Walk them in order when you are stuck: the question tells you what the branch is about, and the first check tells you where to look.

Cost

1. Compute that is on

Are machines running when no work is?

Idle clusters, no auto-termination, over-large warehouses, long auto-stop

First check: Cluster hours vs busy hours

2. Compute that is too big

Is it sized for the job?

Over-sized driver or nodes, autoscale max, memory-optimised without need

First check: CPU and memory use against size

3. Compute that is the wrong kind

Right tool for the work?

Interactive for scheduled jobs, classic vs serverless, no spot, old generations

First check: Job type against compute type

4. Queries that waste

Do queries read or compute more than they need?

Full scans, SELECT *, cartesian joins, spill, recompute

First check: Bytes scanned vs bytes used

5. Storage that grows

Why is stored data growing?

Small files, old versions, log growth, duplicate copies, unused tables

First check: Storage by table and age

6. Refresh that is too frequent

Does anyone need it this fresh?

Over-frequent schedules, dashboards, subscriptions, continuous pipelines for batch

First check: Refresh rate vs read rate

7. Data that moves

What crosses a boundary?

Cross-region, cross-cloud, egress, BI tools pulling extracts

First check: Network and egress lines

8. Spend nobody explains

Who owns this?

Untagged resources, orphaned objects, spikes, new unattributed workloads

First check: Share of cost without an owner

Where the money goes

The five lines on every data bill

Whatever the vendor, the invoice breaks into the same kinds of line. Knowing the line tells you which lever to pull.

Where a typical data bill goes (illustrative shape, not a measurement)Compute62%Storage14%6%12%6%Network 6%, Platform services 12%, Other 6%.Start with idle and oversized compute, then wasteful queries.

The five lines of a bill. The split varies by team; use the shape, not the numbers.

Cost lineWhat you are charged forMain driverBiggest leverVerdict
ComputeMachine time (VMs, units of processing, warehouses)Hours on, size, countSwitch off when idle; right-size; make jobs fasterUsually the biggest line; start here
StorageBytes kept, including old versions and logsVolume and retentionDelete unused data, compact files, limit historyUsually small; surprising when ignored
Network / egressBytes leaving a region or cloudCross-region reads, extractsCo-locate compute and dataSmall until it is not
Platform servicesOrchestration, catalogs, AI function calls, vector search, serving endpointsCalls, tokens, uptimeCap, cache, scale to zeroGrowing fast; watch it
People and timeEngineers fixing, re-running, reconcilingRework and firefightingFewer incidents, earlier checksOften larger than the cloud bill

Compute that is on

Idle is the most common waste

A cluster that waits for work still bills. Interactive clusters left on overnight, warehouses that auto-stop after an hour, jobs that keep a cluster warm between runs: all pay for nothing.

One day on a cluster that never switches off (each block = 1 hour)00h03h06h09h12h15h18h21hbusy (6 h)idle but billed (18 h)Billed for 24 hours, working for 6. An idle timeout of a few minutes removes most of the pink.

Idle compute: the same bill for busy and idle hours. Illustrative example.

PatternWhy it costsFixVerdict
No auto-terminationCluster runs until someone remembersSet a short idle timeoutAlways fix
Long auto-stop on warehousesPays for the whole idle window each timeShorten to minutesPursue
Interactive cluster running scheduled jobsInteractive rates for non-interactive work, and it stays onMove to job computePursue
Always-warm pools nobody usesPay to keep machines readyReduce or removePursue
Continuous pipeline on batch dataStreaming cost for data that arrives hourlyTriggered, scheduled runsPursue
Orphaned clusters of departed usersNobody owns themPolicy: owner required, auto-cleanAlways fix

Compute that is too big or wrong

Right-size, then right-kind

Bigger machines are not faster unless the job can use them. Look at what the machines actually did before paying for more.

PatternSignalFixVerdict
Over-sized nodes or driverCPU and memory mostly idleSmaller instances, smaller driverPursue
Autoscale maximum far above needScales out and sits idleLower the capPursue
Memory-optimised without memory pressurePays a premium for unused memoryGeneral purposePursue
Single node running a distributed workloadSlow, and the long run costs moreRight-size workersPursue
No spot instances for fault-tolerant workPays full priceSpot workers, on-demand driverPursue for retry-safe jobs
Spot used with retry stormsInterruptions rerun workMix on-demand, checkpointPursue
Old runtime or instance generationSlower per dollarUpgradePursue
Photon-style acceleration without benefitHigher unit price, no speed gainMeasure; turn off where it does not payMeasure first

Queries that waste

Performance waste is cost waste

Every wasted byte scanned or shuffled is paid for twice: in compute time now, and in a slower finish. The performance page covers the mechanics; here is how it appears on the bill.

PatternWhy it costsFixVerdict
Recurring full table scansSame bytes read again and againLayout for the common filter; precomputePursue first
SELECT * on wide tablesReads columns nobody needsName the columnsPursue
Loops running a query or action per passFixed start-up cost multipliedOne set-based queryAlways fix
Dashboards hitting raw (bronze) tablesHeavy query on every openPoint at goldPursue
Spill and skewMachines held for the stragglerFix partitions and keysPursue
Cartesian joins at scaleExplosive workAdd the join conditionAlways fix

Storage that grows

Small files, old versions and forgotten tables

Storage is cheap per byte, but table maintenance decides whether storage stays small and fast. Neglected tables get slower and larger together.

PatternWhy it costsFixVerdict
Small filesMore requests, more tasks, slower readsCompact regularlyPursue
Vacuum / clean-up overdueDeleted files still stored and billedSchedule clean-upPursue
Over-long history retentionOld versions kept for nothingRetain what you actually time-travelPursue
Change feed on with no consumerExtra data written for nobodyTurn offPursue
Maintenance run too oftenCompaction thrash rewrites files again and againRun on need, not on a timerPursue
Tables and views with no readersBuilt and refreshed for nobodyRetire themPursue

Refresh that is too frequent

Freshness is a product decision

Every refresh is a cost. The question is not 'can we refresh every minute?' but 'who reads it that often?'

A job that costs $2 per run, per year (illustrative)Monthly$24Weekly$104Daily$730Hourly$17,520Bar length is a square-root scale so the small ones stay visible. Ask how often before asking how fast.

Frequency multiplies everything. Same job, same $2: monthly is $24 a year, hourly is $17,520.

PatternWhy it costsFixVerdict
Cron more frequent than data changesRuns on unchanged dataMatch to arrivalPursue
Dashboard auto-refresh too fastQueries run whether anyone watchesSlower refresh; cachePursue
Subscriptions emailed too oftenSame query each sendReduce frequencyPursue
Dashboards nobody opensRefresh with no readerRetire or pauseAlways fix
Query per viewer fan-outN viewers, N queriesShared cached resultPursue
BI tool direct query on heavy dataEach click is a warehouse queryImport or aggregate in goldPursue

Data that moves

Egress and cross-boundary reads

Moving data between regions or clouds costs per byte and adds latency. Keep compute near data.

PatternWhy it costsFixVerdict
Compute and storage in different regionsEgress on every readCo-locatePursue
Data in several cloudsEgress and duplicate copiesPick a home; share, do not copyPursue
Large extracts to external toolsPays egress and loses controlQuery in placePursue
Chatty orchestrationMany tiny invocations from outsideBatch callsPursue

Spend nobody explains

Anomalies, tags and ownership

You cannot manage a cost you cannot attribute. Tags and owners turn a bill into a conversation.

PatternSignalFixVerdict
No cost tagsSpend cannot be assigned to a teamRequired tags by policyAlways fix
New workload with no ownerSudden unattributed lineAssign an owner on creationPursue
Day-over-day spikeOne day far above normalAlert and trace the jobPursue
Sustained shift upwardA new baseline after a changeReview the changePursue
One user dominates spendRunaway personal computeQuota and reviewPursue
Seasonality ignoredWeekly cycle triggers false alarmsCompare to the same weekdayTune alerts

In practice

CostSherlock rules for this topic

CostSherlock is our open set of cost and performance rules. Each rule below has a plain meaning here; the rule id is its name in CostSherlock. They are grouped by the same branches as the mind map, so a finding tells you which branch to open.

Compute that is on

CostSherlock ruleWhat it catches, in plain words
cluster_no_autoterminationNo idle shutdown
uc_cluster_idle_no_workloadCluster up with no work
uc_warehouse_auto_stop_too_longWarehouse waits too long to stop
uc_cluster_long_lived_interactiveInteractive cluster kept for days
uc_interactive_cluster_for_scheduled_jobScheduled job on an interactive cluster
uc_warehouse_dormant_with_provisioned_clustersWarehouse idle but still provisioned
dlt_continuous_for_batch_workloadContinuous pipeline for batch data

Compute that is too big or wrong

CostSherlock ruleWhat it catches, in plain words
uc_cluster_oversized_for_workloadCluster too large for the work
oversized_autoscale_maxAutoscale ceiling too high
uc_cluster_oversized_driverDriver bigger than needed
uc_cluster_memory_optimized_unusedMemory-optimised, memory unused
uc_cluster_no_spot_workersNo spot workers where safe
uc_spot_cluster_retry_stormSpot interruptions rerunning work
uc_cluster_old_instance_generationOld instance generation
uc_photon_overhead_negative_roiAcceleration costs more than it saves
uc_oversized_warehouse_low_utilizationBig warehouse, low use

Queries that waste

CostSherlock ruleWhat it catches, in plain words
uc_query_full_scan_recurringRecurring full scans
uc_select_star_at_scaleSELECT * on large data
uc_query_cartesian_join_at_scaleCartesian join at scale
uc_query_spill_to_disk_recurringRecurring spill
uc_query_recompute_no_cachingRecompute without caching
code_spark_action_inside_loopAction inside a loop
collect_in_for_loopcollect() in a loop

Storage that grows

CostSherlock ruleWhat it catches, in plain words
too_many_filesToo many small files
uc_vacuum_overdueClean-up overdue
delta_vacuum_overdueVacuum overdue
delta_log_growth_without_checkpointTransaction log growing unchecked
uc_cdf_enabled_no_consumerChange feed with no reader
uc_optimize_thrash_tableCompaction thrash
uc_mv_no_readersMaterialised view nobody reads
uc_delta_time_travel_unusedHistory kept but never used

Refresh that is too frequent

CostSherlock ruleWhat it catches, in plain words
cron_too_frequentSchedule too frequent
uc_dashboard_auto_refresh_too_frequentDashboard refreshes too often
uc_dashboard_subscription_over_frequentSubscriptions too frequent
uc_dashboard_nobody_opensDashboard nobody opens
uc_dashboard_query_per_viewer_fan_outA query per viewer
uc_external_bi_tool_direct_query_heavyBI tool querying directly and heavily
adf_trigger_too_frequentPipeline trigger too frequent

Data that moves

CostSherlock ruleWhat it catches, in plain words
azure_excessive_egressExcessive egress
uc_external_locations_multi_cloudLocations across clouds
uc_external_locations_region_mixedLocations across regions
adf_data_movement_dominates_pipeline_costData movement dominates the pipeline

Spend nobody explains

CostSherlock ruleWhat it catches, in plain words
uc_cluster_no_cost_tagsNo cost tags
cost_anomaly_day_spikeDaily spike
cost_anomaly_sustained_shiftSustained shift
cost_anomaly_new_unattributed_workloadNew unattributed workload
cost_anomaly_dow_seasonality_awareWeekly pattern considered
uc_workspace_runaway_cost_by_userOne user runs away with spend
azure_orphaned_managed_diskOrphaned disk
dbx_orphaned_vm_after_cluster_terminatedVM left after cluster ended

FAQ

Questions people ask

Where do most data platform bills leak?

Idle compute, over-sized compute, and the same expensive query or refresh repeated for no reader. Storage and network are usually smaller lines, but neglected tables slow everything and quietly grow.

Is a faster job always cheaper?

Usually, because machines are held for less time. Paying for a much larger cluster to finish sooner can cost more if the job does not scale with it. Measure cost per run, not just run time.

How often should a pipeline run?

As often as someone acts on the result, and no more. Ask who reads it and what they do differently with a fresher answer.

Does serverless always save money?

No. It removes start-up waits and idle waste for spiky work, and can cost more for steady heavy work. Compare on your own pattern.

What is the first thing to do about cost?

Tag everything with an owner, then look at the top ten items by spend. You cannot fix what nobody owns.

Are the sample figures real?

They are illustrative arithmetic to show how frequency multiplies a cost, not quotes. Prices differ by vendor and contract.

Related

Where it connects

Performance fundamentals · Processing fundamentals · Security fundamentals · Storage fundamentals · Database families · Where data engineering time goes · Who maintains your tables · All data engineering

The numbers and thresholds on this page are rules of thumb and orders of magnitude to build intuition, not benchmarks. Check your platform's current documentation and measure your own workload.