Home › Learn › Databricks interview prep › UniForm

Databricks interview prep · data engineering

How UniForm works inside

How Delta UniForm generates Iceberg metadata, why column mapping and field IDs are required, and the consequences and alternatives.

Maintenance commandsRow-level securityCompute choiceJoinsSkew and shuffleAQE, Photon, POCost and performance leversDeclarative and semanticAsset BundlesLakeflow ConnectUniFormVector Searchdbt and ODCSEgress and ingressMaintenance in picturesClustering and compactionMaintenance strategySlow writesBackup and recoveryTaggingQuestion bank

Defaults, limits and preview status change often. Check the current Databricks docs before quoting a number.

Delta commitAsync step reads new logentriesWrites Iceberg manifestsand metadata.jsonIceberg engines read thesame Parquet

One copy of data, two sets of metadata.

The idea

One copy of data, two sets of metadata

Delta and Iceberg both store Parquet. Only the metadata differs: Delta uses a JSON log; Iceberg uses metadata files, manifest lists and manifests. UniForm keeps Delta as the source of truth and writes Iceberg metadata beside it.

Step by step

What happens on each commit

  1. A Delta commit happens (write, MERGE, OPTIMIZE).
  2. After it, asynchronously, a conversion step reads the new log entries.
  3. It translates them into Iceberg structures (table below).
  4. It writes new files in a metadata/ folder next to _delta_log/: manifests, a manifest list and a new metadata.json.
  5. The catalog points readers at the latest metadata.json; Unity Catalog exposes an Iceberg REST endpoint.
  6. Trino, Snowflake, Athena, Flink and others read the Parquet files using that metadata.
DeltaIceberg
Table version (commit N)A new snapshot
add file action with statisticsData file entry in a manifest, with column min, max and null counts
remove file actionFile marked deleted in the new snapshot
Schema and column mappingIceberg schema with field IDs
Partition columns or clusteringPartition spec
Table propertiesProperties in metadata.json

Why

Column mapping and field IDs

Iceberg identifies columns by numeric field ID, not name. Plain Delta Parquet files usually carry no IDs. So UniForm requires Delta column mapping, which gives each column a stable ID that is also written into the Parquet files. The table must also be in an Iceberg-compatibility mode, which blocks Delta features Iceberg cannot represent. For example deletion vectors must be off or purged; check the current rules.

In practice

What follows

PropertyConsequence
Single copyNo duplicate storage, no sync job
Generated metadataA small extra write per commit
AsynchronousIceberg readers can lag the latest Delta commit briefly; eventually consistent
One-directionalDelta writes; Iceberg clients read only
Time travelIceberg snapshots map to Delta versions, limited by retained history; VACUUM affects it
StatisticsIceberg skipping uses stats carried over from Delta; no stats, no skipping
MaintenanceOPTIMIZE and VACUUM on Delta make new commits, which make new Iceberg snapshots
Feature gapsFeatures with no Iceberg equivalent restrict what you can enable

Compare

Related approaches

ApproachHow it worksTrade-off
UniFormDelta writes; Iceberg metadata generatedOne copy, Delta-first, read-only for Iceberg
Apache XTableOpen-source metadata translator you run yourselfSame idea, you schedule it
Managed Iceberg tablesTable is natively IcebergIceberg is primary; Delta features unused
Copy to a second tableDuplicate dataDouble cost, stale copies
Delta SharingLive data over a protocolFor cross-organization sharing

See Delta vs Iceberg and who owns the catalog.

Practice

Questions to rehearse

  1. What does UniForm duplicate, and what not?
  2. Why does it need column mapping?
  3. Is the Iceberg metadata updated synchronously?
  4. Can an Iceberg engine write to a UniForm table?
  5. What happens to Iceberg readers after OPTIMIZE or VACUUM?
  6. Why might an Iceberg engine scan more than Spark on the same table?
  7. UniForm or managed Iceberg tables?
In an interview, say: UniForm keeps Delta as the writer. After each commit it translates the log into Iceberg manifests and metadata.json beside the data, so the same Parquet serves both formats. It needs column mapping, is read-only and slightly delayed for Iceberg readers.

FAQ

Questions people ask

Does UniForm copy my data?

No. It writes only metadata; the Parquet files are shared.

Why can Iceberg engines skip less?

They rely on statistics carried over from Delta.

Next

Where to go next

All Databricks interview topics · All data engineering · Databricks on this site