Home › Key concepts › Open table formats and ACID

Key concept · Walk (intermediate) · the same on every platform

Open table formats and ACID

Delta Lake, Apache Iceberg and Apache Hudi make files in storage behave like database tables.

What it is

A table format adds a transaction log on top of Parquet files. That gives ACID transactions (all-or-nothing writes that readers never see half-done), updates and deletes, schema changes, and time travel to read the table as it was yesterday.

See it happen: Delta Lake internals, recorded (simulation), recorded from a real run.

Why it matters

It turns a cheap data lake into something you can trust like a warehouse, and lets many engines read the same tables.

An example

A failed job wrote half its files, but readers never see them because the commit was never written to the log.

The common mistake

Writing plain Parquet folders by hand and then building your own fragile locking and cleanup.

Where you meet it on each platform

The idea stays the same; only the product name changes.

JobAWSAzureGoogle CloudDatabricksOpen source
Open table formatfiles that behave like a database table, with historyApache Iceberg (S3 Tables)Delta Lake in OneLakeIceberg on BigLakeDelta LakeDelta Lake or Iceberg

Related ideas

Where it shows up

Part of the Walk level on these paths: Data Engineering (storing data).