Home › Key concepts › The small-files problem
Key concept · Walk (intermediate) · the same on every platform
The small-files problem
Thousands of tiny files make every query slow.
What it is
Every file has a cost to list, open and read. Streaming and frequent small writes create thousands of files of a few kilobytes. Engines then spend more time opening files than reading data. The fix is compaction: rewriting many small files into fewer large ones (around 128 MB to 1 GB).
See it happen: Delta Lake internals: 300 tiny files, then OPTIMIZE (simulation), recorded from a real run.
Why it matters
It is one of the most common and most expensive mistakes in data lakes, and it hides quietly until queries crawl.
An example
A table with 200,000 files averaging 40 KB is compacted to 80 files of 100 MB, and the same query runs in seconds instead of minutes.
The common mistake
Over-partitioning, so every partition holds a handful of tiny files.
Where you meet it on each platform
The idea stays the same; only the product name changes.
| Job | AWS | Azure | Google Cloud | Databricks | Open source |
|---|---|---|---|---|---|
| Open table formatfiles that behave like a database table, with history | Apache Iceberg (S3 Tables) | Delta Lake in OneLake | Iceberg on BigLake | Delta Lake | Delta Lake or Iceberg |
| Batch computeprocess a lot of data at once | AWS Glue / EMR (Spark) | Fabric Spark | Dataproc (Spark) | Databricks jobs (Spark) | PySpark or Polars |
Related ideas
Where it shows up
Part of the Walk level on these paths: Data Engineering (storing data).