Home › Key concepts › Separating storage and compute
Key concept · Walk (intermediate) · the same on every platform
Separating storage and compute
Keep data in one place and rent the engines that read it only when needed.
What it is
In older systems the disks and the processors came together. In the cloud, data sits in object storage and compute clusters start, read it, and stop. You pay for storage all the time and for compute only while it runs.
Why it matters
Most cloud data cost is compute. Clusters left running, or sized too large, are where bills go wrong.
An example
A nightly job starts 8 machines, runs for 20 minutes and shuts down; the data stays in storage at a few dollars a month.
The common mistake
Leaving an all-purpose cluster running all weekend for one dashboard.
Where you meet it on each platform
The idea stays the same; only the product name changes.
| Job | AWS | Azure | Google Cloud | Databricks | Open source |
|---|---|---|---|---|---|
| Object storagecheap files in the cloud, the floor everything stands on | Amazon S3 | ADLS Gen2 / OneLake | Cloud Storage | Unity Catalog volumes on cloud storage | Local disk or MinIO |
| Batch computeprocess a lot of data at once | AWS Glue / EMR (Spark) | Fabric Spark | Dataproc (Spark) | Databricks jobs (Spark) | PySpark or Polars |
| Costsee what each job and query costs, then cut it | Cost Explorer + CUR | Cost Management | Billing export to BigQuery | Billing system tables | Query plans and run times |
Related ideas
Where it shows up
Part of the Walk level on these paths: Data Engineering (storing data).