Home › Key concepts › The small-files problem

Key concept · Walk (intermediate) · the same on every platform

The small-files problem

Thousands of tiny files make every query slow.

What it is

Every file has a cost to list, open and read. Streaming and frequent small writes create thousands of files of a few kilobytes. Engines then spend more time opening files than reading data. The fix is compaction: rewriting many small files into fewer large ones (around 128 MB to 1 GB).

Why it matters

It is one of the most common and most expensive mistakes in data lakes, and it hides quietly until queries crawl.

An example

A table with 200,000 files averaging 40 KB is compacted to 80 files of 100 MB, and the same query runs in seconds instead of minutes.

The common mistake

Over-partitioning, so every partition holds a handful of tiny files.

Where you meet it on each platform

The idea stays the same; only the product name changes.

JobAWSAzureGoogle CloudDatabricksOpen source
Open table formatfiles that behave like a database table, with historyApache Iceberg (S3 Tables)Delta Lake in OneLakeIceberg on BigLakeDelta LakeDelta Lake or Iceberg
Batch computeprocess a lot of data at onceAWS Glue / EMR (Spark)Fabric SparkDataproc (Spark)Databricks jobs (Spark)PySpark or Polars

Related ideas

Where it shows up

Part of the Walk level on these paths: Data Engineering (storing data).