Home › Key concepts › Separating storage and compute

Key concept · Walk (intermediate) · the same on every platform

Separating storage and compute

Keep data in one place and rent the engines that read it only when needed.

What it is

In older systems the disks and the processors came together. In the cloud, data sits in object storage and compute clusters start, read it, and stop. You pay for storage all the time and for compute only while it runs.

Why it matters

Most cloud data cost is compute. Clusters left running, or sized too large, are where bills go wrong.

An example

A nightly job starts 8 machines, runs for 20 minutes and shuts down; the data stays in storage at a few dollars a month.

The common mistake

Leaving an all-purpose cluster running all weekend for one dashboard.

Where you meet it on each platform

The idea stays the same; only the product name changes.

JobAWSAzureGoogle CloudDatabricksOpen source
Object storagecheap files in the cloud, the floor everything stands onAmazon S3ADLS Gen2 / OneLakeCloud StorageUnity Catalog volumes on cloud storageLocal disk or MinIO
Batch computeprocess a lot of data at onceAWS Glue / EMR (Spark)Fabric SparkDataproc (Spark)Databricks jobs (Spark)PySpark or Polars
Costsee what each job and query costs, then cut itCost Explorer + CURCost ManagementBilling export to BigQueryBilling system tablesQuery plans and run times

Related ideas

Where it shows up

Part of the Walk level on these paths: Data Engineering (storing data).