Home › Key concepts › Where data costs come from
Key concept · Run (advanced) · the same on every platform
Where data costs come from
Scanning, shuffling, skew and idle clusters: the four usual suspects.
What it is
Scan cost grows with the data a query reads. Shuffle is moving data between machines for joins and group-bys. Skew is when one machine gets most of the work, so everyone waits for it. Idle clusters cost money while doing nothing.
See it happen: Performance stories: the slow job, the clue, the fix, measured (simulation), recorded from a real run.
Why it matters
Knowing these four lets you cut most bills without new tools.
An example
One customer with 40% of all orders makes one task run for an hour while 199 others finish in a minute; salting the key fixes it.
The common mistake
Buying bigger clusters to fix a skew problem.
Where you meet it on each platform
The idea stays the same; only the product name changes.
| Job | AWS | Azure | Google Cloud | Databricks | Open source |
|---|---|---|---|---|---|
| Costsee what each job and query costs, then cut it | Cost Explorer + CUR | Cost Management | Billing export to BigQuery | Billing system tables | Query plans and run times |
| Batch computeprocess a lot of data at once | AWS Glue / EMR (Spark) | Fabric Spark | Dataproc (Spark) | Databricks jobs (Spark) | PySpark or Polars |
Related ideas
Where it shows up
Part of the Run level on these paths: Data Engineering (running data).