Home › Learn › Databricks interview prep › Clustering and compaction

Databricks interview prep · data engineering

Compaction mechanics, and partitioning vs Z-order vs liquid clustering

Who really compacts (cloud or Databricks), snappy versus compaction, when each mechanism triggers, and how liquid clustering differs inside from Z-order and partitioning.

Maintenance commandsRow-level securityCompute choiceJoinsSkew and shuffleAQE, Photon, POCost and performance leversDeclarative and semanticAsset BundlesLakeflow ConnectUniFormVector Searchdbt and ODCSEgress and ingressMaintenance in picturesClustering and compactionMaintenance strategySlow writesBackup and recoveryTaggingQuestion bank

Defaults, limits and preview status change often. Check the current Databricks docs before quoting a number.

Partitioning: a folder per valuecountry=IN/ country=US/ country=UK/Fixed at creation. High-cardinalitycolumns make millions of tiny files.INUSUKDEZ-order: one curve through the dataOrders rows by interleaving the bitsof the chosen columns. Re-run tocluster new data; keys chosen each run.Liquid clustering: groups that evolveKeys live in table metadata. OPTIMIZEclusters only what is not yet clustered;keys can change without a full rewrite.

Three ways to lay data out so a filter can skip files. Only partitioning is a physical folder structure.

Compaction

Is it the cloud or Databricks? What about snappy?

QuestionAnswer
Who compacts: the cloud or Databricks?Delta's engine does it, not the cloud. ADLS Gen2, S3 and GCS store objects and never merge them. Only some cloud-managed Iceberg services (AWS Glue, S3 Tables) compact for you; Azure has no managed equivalent for Delta.
Is snappy a compaction?No. Snappy and zstd are compression codecs that shrink the data inside one Parquet file. Compaction merges many files into fewer. They are independent: OPTIMIZE rewrites files using whatever codec the table writes with (snappy has long been the default; newer runtimes may default to zstd, so check).
What percentage of compaction does each mechanism do?There is no such split. Each mechanism is a rule with thresholds, not a share. Measure your own table: compare files per partition and average file size before and after.

Defaults below are from documentation and memory. Check your runtime.

MechanismWhen it kicks inWhat it produces
Optimized writesOn every write that has the setting onFiles of roughly 128 MB, by shuffling before the write
Auto compactionAfter a write commits, when a partition has at least 50 small files (the default for minNumFiles)Files up to about 128 MB; runs inside the writing job
OPTIMIZEWhen you or a job run it; bins files below the minimum sizeFiles toward about 1 GB (Databricks auto-tunes the target by table size)
Predictive OptimizationWhen its cost model judges it worth it; managed tables onlyThe same as OPTIMIZE, scheduled for you
Cloud-managed Iceberg compactionService-defined; Iceberg tables onlyService-defined

Layouts

Partitioning, Z-order and liquid clustering compared

PartitioningZ-orderLiquid clustering
What it isOne folder per valueRows ordered along a space-filling curve (interleaved bits)Rows ordered along a curve (documented as a Hilbert curve) with keys held in table metadata
Where the layout livesFolder structureIn the files, after each OPTIMIZE ZORDERIn the files and the log; the log knows which files are clustered
Change the keysRewrite the whole tableName different columns next run, rewrites dataALTER TABLE … CLUSTER BY; new data follows the new keys, OPTIMIZE re-clusters old files over time
Incrementaln/aLimited: costs grow as you re-runYes: OPTIMIZE only touches files not yet clustered
High-cardinality columnDisaster: millions of foldersWorks, up to about four columnsWorks
SkewOne huge partitionPartlyHandled better, files are sized by data not by value
Concurrent writesConflicts within a partitionNormalRow-level concurrency possible
Can combineWith Z-order inside a partitionWith partitioningNot with partitioning or Z-order on the same table
Use whenLow-cardinality column, tables over about 1 TB, time-based retention by folderExisting tablesNew tables; the default choice

How liquid clustering works inside (from documentation; I could not measure it, see the other page): the clustering columns are stored in the table metadata. When data is written, or when OPTIMIZE runs, rows are ordered along the curve and cut into files so that each file covers a compact region of the clustering columns. The log records per-file minimum and maximum values for those columns, and each file carries a mark saying it is clustered. OPTIMIZE reads those marks and rewrites only unmarked files, which is why it is incremental. Changing the keys only changes the metadata; old files are re-clustered later or all at once with a full OPTIMIZE.

Why Z-order is weaker: the keys are not remembered by the table, so every run must name them, and the cost of keeping data clustered grows with the table because earlier work is not tracked as precisely.

Practice

Questions to rehearse

  1. Is compaction a cloud feature or a Databricks feature?
  2. Snappy versus compaction: what is the difference?
  3. When does auto compaction start?
  4. Why can liquid clustering change keys without a rewrite?
  5. Why is partitioning on a high-cardinality column harmful?
  6. Z-order versus liquid clustering: three differences.
In an interview, say: Compaction merges files, a codec compresses inside a file. Liquid clustering keeps its keys in the table and clusters incrementally, which is why it replaces partitioning and Z-order for new tables.

FAQ

Questions people ask

Can I use liquid clustering with partitioning?

No.

Does snappy reduce the number of files?

No. It only compresses the contents of each file.

Next

Where to go next

All Databricks interview topics · All data engineering · Databricks on this site