Delta, Iceberg, or both? Open table formats are settled

An open table format is Parquet files plus a log that turns a folder into a real table, with transactions, time travel, schema changes and safe concurrent writes. The move to open formats has settled. Delta Lake and Apache Iceberg do the same job with different bookkeeping, and today you can often keep one copy of the data and let readers of both kinds use it.

Every few months a data team holds the same meeting. Someone has read that Iceberg is winning. Someone else points out that the whole platform runs on Delta. A third person mentions Hudi. An hour later nothing is decided, and the tables stay exactly where they were.

The meeting is arguing about the smaller question. The big one was settled years ago: your tables should live as open files in your own storage, readable by more than one engine, and not locked inside one vendor's database.

This post is part of our series on what has settled, after building agents on a moving ship. It explains what a table format actually does, using numbers recorded from a real Delta Lake run, then compares Delta and Iceberg, and shows how to read both without picking a side.

Loose pages and a binder

A data lake starts as a folder of Parquet files. Parquet is a column-based file format: compact, fast to scan, and readable by almost every engine. But a folder of Parquet files is a pile of loose pages. Nothing says which pages belong to today's table, which were half-written by a job that crashed, or which were replaced last week.

A FOLDER OF PARQUET FILESAN OPEN TABLE FORMATwhich pages are today's table?which were half-written by a crash?The logtable of contents:v1 = files a, bv2 = + c, dv3 = - b, + esame Parquet files, plus a list ofexactly which ones make each version
A table format doesn't change the pages. It adds the table of contents.

An open table format puts those pages in a binder with a table of contents. The pages are still ordinary Parquet files. The table of contents is a log, or a set of metadata files, sitting next to them. It lists exactly which files make up the table at each version. A reader opens the table of contents first and reads only the pages it names.

That is the whole trick. Delta Lake, Apache Iceberg and Apache Hudi are three ways of keeping that table of contents. Our concept page on open table formats has the short version.

What the log adds

Once there is a table of contents, four things become possible that a plain folder could never do safely.

what the log makes possible on plain Parquet files1Transactionsa write fully happensor not at all2Time travelread the tableas it was at v13Schema changesadd a columnwithout rewriting4Safe writerstwo jobs at once,no lost rows
The four things a log adds. All four come from one idea: readers trust the list, not the folder.
  1. Transactions. A write either fully happens or doesn't happen at all. In our recorded Delta run (Spark 3.5.1 with Delta Lake 3.2.0, on a laptop), an insert first wrote two Parquet files, one per Spark task, and only then wrote commit 1 naming them. Until that commit file existed, nobody could see the rows. A job that crashes halfway leaves files no commit names, and readers never see them.
  2. Time travel. Old files are not deleted when rows change; the log simply stops listing them. In the recording, reading version 1 meant replaying only commits 0 and 1, and yesterday's table came back exactly, including a row that had since been deleted.
  3. Schema changes. The log records the table's columns. Adding a column is a new entry in the log, not a rewrite of every file.
  4. Safe concurrent writers. Two writers race to create the next commit file, and only one can win. In the recording, two writers that both appended rows both succeeded: the loser checked that the winner had only added rows and took the next version number. Two writers that updated the same row did not: the second failed with a ConcurrentAppendException instead of silently overwriting the first.

These four are why a cheap lake can be trusted like a warehouse, and why the medallion layers can be rebuilt with confidence.

Why this has settled

Three things make open table formats a plank, not a crate.

No serious new platform today asks you to load your data into a closed format first. That is what "settled" means.

Delta and Iceberg: same idea, different bookkeeping

Delta and Iceberg solve the same problem. The difference is where the table of contents lives and who keeps it.

DELTA LAKEAPACHE ICEBERGthe _delta_log folder000.json001.json002.json010 checkpointParquet data filescommit = create the nextnumbered file, all or nothingCatalogpoints to currentMetadata filethis versionManifestslists of filesParquet data filescommit = the catalogswaps its pointerto a new metadata file
Same pages underneath. Delta keeps the table of contents in a folder; Iceberg keeps it in a tree, with a catalog holding the bookmark.

Delta Lake keeps a folder called _delta_log next to the data. Each commit is a numbered JSON file listing the files added and removed. Replaying hundreds of JSON files would be slow, so every 10 commits Delta writes a checkpoint, one Parquet file holding the whole state, and readers start from there. A commit happens when a new numbered file is created, and creating it is all-or-nothing.

Apache Iceberg keeps a tree of metadata files: a table metadata file, which points to a list of manifests, which point to the data files. A catalog, a small service that knows where each table's current metadata file is, holds a pointer to the latest one. A commit happens when the catalog swaps that pointer.

Delta LakeApache Iceberg
Table of contentsnumbered commits in _delta_logmetadata tree, pointer held by a catalog
Strongest withSpark and Databricks, Microsoft Fabricmany engines at once: Trino, Flink, Snowflake, AWS, Google Cloud
Known fordeep Spark integration, change data feed, deletion vectorshidden partitioning, changing the partition scheme without rewriting data
Clean-up jobsOPTIMIZE and VACUUMrewriting data files, expiring snapshots

The honest summary: if most of your work runs on Databricks or Fabric, Delta is the natural home. If many engines from many vendors need to write the same tables, Iceberg's catalog-centred design was built for that. The two keep borrowing each other's best ideas, so the feature gap narrows every year.

Reading both without picking a side

The newer answer to "Delta or Iceberg?" is often "write one, let both kinds of reader in". The goal is one copy of the data, two kinds of readers.

one copy of the data, two kinds of readersOne writere.g. a Spark jobthe same Parquet filesDelta logIcebergmetadataCatalogIceberg RESTinterfaceIceberg engineanother engineUniForm: Delta writes the Iceberg metadata toono second copy, no nightly sync job
Read both: one set of files, two tables of contents, and a catalog that tells every engine where to look.

There are also tools that translate metadata between formats after the fact. Whatever you use, the test is simple: is there one set of Parquet files, with one owner who writes them? If yes, adding a reader is configuration, not a new pipeline.

Hudi, briefly

Apache Hudi came out of Uber and is the third open format. It is strongest at heavy upserts and incremental pulls, meaning "give me only the rows that changed since my last read", and it offers both copy-on-write and merge-on-read tables. It has a loyal following on streaming-heavy platforms. For most new projects the choice is Delta or Iceberg, but a working Hudi platform is no reason to migrate.

Mistakes we see often

  1. Choosing a format by hype. The right format is the one your main engines write best. Follow your platform, not the conference talk.
  2. Never running OPTIMIZE or VACUUM. Every small write adds small files, and old files stay on disk for time travel. In our recording, 300 small files made a one-customer query take 777 ms; after OPTIMIZE compacted them into one file, the same query took 278 ms. The 300 old files stayed on disk until VACUUM removed them. Our Databricks cost leaks post covers small files in more depth.
  3. Letting VACUUM break time travel. VACUUM deletes files no current version needs, once they are older than the retention period, 7 days by default. In the recording, retention was forced to 0 hours, which Delta refuses until a safety check is switched off. Afterwards the log still listed version 4, but reading it failed with a "file does not exist" error. Retention is how far back you can travel. Set it on purpose.
  4. Two copies of the data to serve two engines. A nightly job that copies Delta tables into Iceberg doubles storage, adds a lag, and creates two truths that drift apart. Use UniForm or a shared catalog instead.
  5. Hand-rolled Parquet folders. Writing plain Parquet and building your own locking and clean-up recreates a table format, badly.

Six questions to check your own setup

  1. Which format do your main write engines handle best, and is that the one you use?
  2. Is any table copied into a second format just so another engine can read it?
  3. When did OPTIMIZE or compaction last run on your busiest table?
  4. What is your VACUUM or snapshot retention, and does it match how far back people need to travel?
  5. If two jobs write the same table at once, do you know what happens?
  6. Could a new engine read your tables tomorrow through a catalog, without a new pipeline?

Pick the format your platform writes best, keep one copy, and let the readers come to it.