Home › Key concepts › Where data costs come from

Key concept · Run (advanced) · the same on every platform

Where data costs come from

Scanning, shuffling, skew and idle clusters: the four usual suspects.

What it is

Scan cost grows with the data a query reads. Shuffle is moving data between machines for joins and group-bys. Skew is when one machine gets most of the work, so everyone waits for it. Idle clusters cost money while doing nothing.

Why it matters

Knowing these four lets you cut most bills without new tools.

An example

One customer with 40% of all orders makes one task run for an hour while 199 others finish in a minute; salting the key fixes it.

The common mistake

Buying bigger clusters to fix a skew problem.

Where you meet it on each platform

The idea stays the same; only the product name changes.

JobAWSAzureGoogle CloudDatabricksOpen source
Costsee what each job and query costs, then cut itCost Explorer + CURCost ManagementBilling export to BigQueryBilling system tablesQuery plans and run times
Batch computeprocess a lot of data at onceAWS Glue / EMR (Spark)Fabric SparkDataproc (Spark)Databricks jobs (Spark)PySpark or Polars

Related ideas

Where it shows up

Part of the Run level on these paths: Data Engineering (running data).