Home › Key concepts › Partitioning and clustering
Key concept · Walk (intermediate) · the same on every platform
Partitioning and clustering
Arrange data so a query can skip what it does not need.
What it is
Partitioning splits a table into folders by a column such as date, so a query for one day reads one folder. Clustering (also called Z-ordering or liquid clustering) sorts data within files so filters on other columns can skip files too.
See it happen: Problem Lab: partitioning too finely turns 600 files into thousands (simulation), recorded from a real run.
Why it matters
Skipping data is the biggest single lever on query speed and cost.
An example
Events partitioned by event_date: 'yesterday's events' reads 1 of 900 folders.
The common mistake
Partitioning by a column with millions of values, such as customer id, creating millions of tiny folders.
Where you meet it on each platform
The idea stays the same; only the product name changes.
| Job | AWS | Azure | Google Cloud | Databricks | Open source |
|---|---|---|---|---|---|
| Open table formatfiles that behave like a database table, with history | Apache Iceberg (S3 Tables) | Delta Lake in OneLake | Iceberg on BigLake | Delta Lake | Delta Lake or Iceberg |
| Warehousetables built for analysis, fast and governed | Amazon Redshift | Fabric Warehouse | BigQuery | Databricks SQL warehouse | DuckDB or Postgres |
Related ideas
Where it shows up
Part of the Walk level on these paths: Data Engineering (storing data).