Home › Learn › Databricks interview prep › Clustering and compaction
Databricks interview prep · data engineering
Compaction mechanics, and partitioning vs Z-order vs liquid clustering
Who really compacts (cloud or Databricks), snappy versus compaction, when each mechanism triggers, and how liquid clustering differs inside from Z-order and partitioning.
Defaults, limits and preview status change often. Check the current Databricks docs before quoting a number.
Three ways to lay data out so a filter can skip files. Only partitioning is a physical folder structure.
Compaction
Is it the cloud or Databricks? What about snappy?
| Question | Answer |
|---|---|
| Who compacts: the cloud or Databricks? | Delta's engine does it, not the cloud. ADLS Gen2, S3 and GCS store objects and never merge them. Only some cloud-managed Iceberg services (AWS Glue, S3 Tables) compact for you; Azure has no managed equivalent for Delta. |
| Is snappy a compaction? | No. Snappy and zstd are compression codecs that shrink the data inside one Parquet file. Compaction merges many files into fewer. They are independent: OPTIMIZE rewrites files using whatever codec the table writes with (snappy has long been the default; newer runtimes may default to zstd, so check). |
| What percentage of compaction does each mechanism do? | There is no such split. Each mechanism is a rule with thresholds, not a share. Measure your own table: compare files per partition and average file size before and after. |
Defaults below are from documentation and memory. Check your runtime.
| Mechanism | When it kicks in | What it produces |
|---|---|---|
| Optimized writes | On every write that has the setting on | Files of roughly 128 MB, by shuffling before the write |
| Auto compaction | After a write commits, when a partition has at least 50 small files (the default for minNumFiles) | Files up to about 128 MB; runs inside the writing job |
| OPTIMIZE | When you or a job run it; bins files below the minimum size | Files toward about 1 GB (Databricks auto-tunes the target by table size) |
| Predictive Optimization | When its cost model judges it worth it; managed tables only | The same as OPTIMIZE, scheduled for you |
| Cloud-managed Iceberg compaction | Service-defined; Iceberg tables only | Service-defined |
Layouts
Partitioning, Z-order and liquid clustering compared
| Partitioning | Z-order | Liquid clustering | |
|---|---|---|---|
| What it is | One folder per value | Rows ordered along a space-filling curve (interleaved bits) | Rows ordered along a curve (documented as a Hilbert curve) with keys held in table metadata |
| Where the layout lives | Folder structure | In the files, after each OPTIMIZE ZORDER | In the files and the log; the log knows which files are clustered |
| Change the keys | Rewrite the whole table | Name different columns next run, rewrites data | ALTER TABLE … CLUSTER BY; new data follows the new keys, OPTIMIZE re-clusters old files over time |
| Incremental | n/a | Limited: costs grow as you re-run | Yes: OPTIMIZE only touches files not yet clustered |
| High-cardinality column | Disaster: millions of folders | Works, up to about four columns | Works |
| Skew | One huge partition | Partly | Handled better, files are sized by data not by value |
| Concurrent writes | Conflicts within a partition | Normal | Row-level concurrency possible |
| Can combine | With Z-order inside a partition | With partitioning | Not with partitioning or Z-order on the same table |
| Use when | Low-cardinality column, tables over about 1 TB, time-based retention by folder | Existing tables | New tables; the default choice |
How liquid clustering works inside (from documentation; I could not measure it, see the other page): the clustering columns are stored in the table metadata. When data is written, or when OPTIMIZE runs, rows are ordered along the curve and cut into files so that each file covers a compact region of the clustering columns. The log records per-file minimum and maximum values for those columns, and each file carries a mark saying it is clustered. OPTIMIZE reads those marks and rewrites only unmarked files, which is why it is incremental. Changing the keys only changes the metadata; old files are re-clustered later or all at once with a full OPTIMIZE.
Why Z-order is weaker: the keys are not remembered by the table, so every run must name them, and the cost of keeping data clustered grows with the table because earlier work is not tracked as precisely.
Practice
Questions to rehearse
- Is compaction a cloud feature or a Databricks feature?
- Snappy versus compaction: what is the difference?
- When does auto compaction start?
- Why can liquid clustering change keys without a rewrite?
- Why is partitioning on a high-cardinality column harmful?
- Z-order versus liquid clustering: three differences.
FAQ
Questions people ask
Can I use liquid clustering with partitioning?
No.
Does snappy reduce the number of files?
No. It only compresses the contents of each file.
Next
Where to go next
All Databricks interview topics · All data engineering · Databricks on this site