Home › Learn › Databricks interview prep › UniForm
Databricks interview prep · data engineering
How UniForm works inside
How Delta UniForm generates Iceberg metadata, why column mapping and field IDs are required, and the consequences and alternatives.
Defaults, limits and preview status change often. Check the current Databricks docs before quoting a number.
One copy of data, two sets of metadata.
The idea
One copy of data, two sets of metadata
Delta and Iceberg both store Parquet. Only the metadata differs: Delta uses a JSON log; Iceberg uses metadata files, manifest lists and manifests. UniForm keeps Delta as the source of truth and writes Iceberg metadata beside it.
Step by step
What happens on each commit
- A Delta commit happens (write, MERGE, OPTIMIZE).
- After it, asynchronously, a conversion step reads the new log entries.
- It translates them into Iceberg structures (table below).
- It writes new files in a
metadata/folder next to_delta_log/: manifests, a manifest list and a newmetadata.json. - The catalog points readers at the latest metadata.json; Unity Catalog exposes an Iceberg REST endpoint.
- Trino, Snowflake, Athena, Flink and others read the Parquet files using that metadata.
| Delta | Iceberg |
|---|---|
| Table version (commit N) | A new snapshot |
| add file action with statistics | Data file entry in a manifest, with column min, max and null counts |
| remove file action | File marked deleted in the new snapshot |
| Schema and column mapping | Iceberg schema with field IDs |
| Partition columns or clustering | Partition spec |
| Table properties | Properties in metadata.json |
Why
Column mapping and field IDs
Iceberg identifies columns by numeric field ID, not name. Plain Delta Parquet files usually carry no IDs. So UniForm requires Delta column mapping, which gives each column a stable ID that is also written into the Parquet files. The table must also be in an Iceberg-compatibility mode, which blocks Delta features Iceberg cannot represent. For example deletion vectors must be off or purged; check the current rules.
In practice
What follows
| Property | Consequence |
|---|---|
| Single copy | No duplicate storage, no sync job |
| Generated metadata | A small extra write per commit |
| Asynchronous | Iceberg readers can lag the latest Delta commit briefly; eventually consistent |
| One-directional | Delta writes; Iceberg clients read only |
| Time travel | Iceberg snapshots map to Delta versions, limited by retained history; VACUUM affects it |
| Statistics | Iceberg skipping uses stats carried over from Delta; no stats, no skipping |
| Maintenance | OPTIMIZE and VACUUM on Delta make new commits, which make new Iceberg snapshots |
| Feature gaps | Features with no Iceberg equivalent restrict what you can enable |
Compare
Related approaches
| Approach | How it works | Trade-off |
|---|---|---|
| UniForm | Delta writes; Iceberg metadata generated | One copy, Delta-first, read-only for Iceberg |
| Apache XTable | Open-source metadata translator you run yourself | Same idea, you schedule it |
| Managed Iceberg tables | Table is natively Iceberg | Iceberg is primary; Delta features unused |
| Copy to a second table | Duplicate data | Double cost, stale copies |
| Delta Sharing | Live data over a protocol | For cross-organization sharing |
See Delta vs Iceberg and who owns the catalog.
Practice
Questions to rehearse
- What does UniForm duplicate, and what not?
- Why does it need column mapping?
- Is the Iceberg metadata updated synchronously?
- Can an Iceberg engine write to a UniForm table?
- What happens to Iceberg readers after OPTIMIZE or VACUUM?
- Why might an Iceberg engine scan more than Spark on the same table?
- UniForm or managed Iceberg tables?
FAQ
Questions people ask
Does UniForm copy my data?
No. It writes only metadata; the Parquet files are shared.
Why can Iceberg engines skip less?
They rely on statistics carried over from Delta.
Next
Where to go next
All Databricks interview topics · All data engineering · Databricks on this site