Home › Key concepts › Partitioning and clustering

Key concept · Walk (intermediate) · the same on every platform

Partitioning and clustering

Arrange data so a query can skip what it does not need.

What it is

Partitioning splits a table into folders by a column such as date, so a query for one day reads one folder. Clustering (also called Z-ordering or liquid clustering) sorts data within files so filters on other columns can skip files too.

Why it matters

Skipping data is the biggest single lever on query speed and cost.

An example

Events partitioned by event_date: 'yesterday's events' reads 1 of 900 folders.

The common mistake

Partitioning by a column with millions of values, such as customer id, creating millions of tiny folders.

Where you meet it on each platform

The idea stays the same; only the product name changes.

JobAWSAzureGoogle CloudDatabricksOpen source
Open table formatfiles that behave like a database table, with historyApache Iceberg (S3 Tables)Delta Lake in OneLakeIceberg on BigLakeDelta LakeDelta Lake or Iceberg
Warehousetables built for analysis, fast and governedAmazon RedshiftFabric WarehouseBigQueryDatabricks SQL warehouseDuckDB or Postgres

Related ideas

Where it shows up

Part of the Walk level on these paths: Data Engineering (storing data).