Home › Learn › How reads and writes work › 1 · Files

Delta Lake and Spark · from zero to advanced

What is on disk: Parquet files and the delta log

A Delta table is a folder of Parquet files plus a transaction log. See a real log (the add action with its min and max statistics, the commit files, a checkpoint) and the real inside of a Parquet file: row groups, column chunks, encodings and the footer.

From zero

A Delta table is a folder of files

There is no database server holding a Delta table. Open the folder and you see two kinds of things: Parquet data files, and a _delta_log folder that says which of those files currently make up the table.

events/ (the table)_delta_log/ (the transaction log)data files part-0000….snappy.parquet000…000.json000…001.json000…004.json000…005.checkpoint.parquet000…005.json000…009.json_last_checkpointa real log with 10 commits (table L)part-00000-….parquet (10,000 rows)part-00001-….parquet (10,000 rows)part-00002-….parquet (10,000 rows)part-00003-….parquet (10,000 rows)part-00004-….parquet (10,000 rows)20 files in the main table TThe log is smalland says whichfiles are live

The folder, drawn from a real table. The log is numbered; the checkpoint is a shortcut.

Measured. The main table has 20 Parquet files and one commit. A second small table made with ten commits shows the checkpoint: Delta wrote 00000000000000000005.checkpoint.parquet and a _last_checkpoint file that says version 5, 10 live files, 16,530 bytes.

Inside a data file

Parquet: columns stored in chunks, with a footer of statistics

Each data file is Parquet. Inside it, data is cut into row groups, and inside each row group every column is its own chunk. At the very end sits a footer with the schema, where each chunk starts, and the minimum and maximum of each column chunk.

PAR1Row group 1 (10,000 rows, 40 column chunks)event_iduser_idcountryamountpayloadf01 … f33late_codeFooterschema,where each columnchunk starts,min and maxPAR1Readers start at the end: read the footer, then jump straight to the column chunks they need.This real file: 562,960 bytes, 1 row group, codec SNAPPY, written by parquet-mr version 1.13.1.

Reading starts at the footer, not the top of the file.

Measured. One real file from the main table: 562,960 bytes holding 10,000 rows in 1 row group with 40 column chunks. Its footer says event_id runs from 100000 to 109999. That one line is what lets a reader skip this whole file when you ask for event_id = 50.

Inside the log

The add action: where the statistics live

Each commit file is plain JSON, one action per line. The important action is add: it names a data file and carries that file's statistics, so the driver can decide which files matter without opening any of them.

{"add": { "path": "part-00000-48860c8e-7919-4949-…parquet", "partitionValues": {}, "size": 564311, "dataChange": true, "stats": { "numRecords": 10000, "minValues": {"event_id": 0, "user_id": 1, "event_date": "2026-01-01", …}, "maxValues": {"event_id": 9999, "user_id": 49998, "event_date": "2026-01-01", …}, "nullCount": {"event_id": 0, "user_id": 0, …} }}}path: which filesize: used to plan tasksnumRecords: lets count(*) beanswered without opening filesmin and max: skip files thatcannot hold the value you wantnullCount: IS NULL filters

A real add action, lightly shortened. This is the line the driver reads.

The same commit also holds a protocol action (this table needs reader version 1, writer version 2), a metaData action (schema, partition columns, table settings), and a commitInfo action recording who did what: operation WRITE, mode Overwrite, isolation level Serializable, written by Apache-Spark/3.5.1 Delta-Lake/3.2.0.

Putting commits together

How a reader works out which files are live

version 0+10 add(a table of 10 files)version 1−1 remove, +1 add(an UPDATE rewrote one file)version 2 …more commitsa checkpoint, for examplev5: all live files savedLive files = every add, minus every file that was later removedA reader loads the latest checkpoint, then replays only the JSON commits after itOld Parquet files stay on disk after a remove. That is what makes time travel possible, until VACUUM deletes them.

Add and remove, replayed in order, give the table's current state.

  • Every add makes a file live. Every remove makes it not live. The current table is what is left after replaying the commits in order.
  • To avoid replaying thousands of small files, Delta writes a checkpoint every few commits: one Parquet file holding the whole live set. A reader loads the latest checkpoint and then only the JSON commits after it.
  • Remove does not delete. The old Parquet file stays in the folder. That is why VERSION AS OF 0 still works after an overwrite, and why storage grows until VACUUM runs (see the write path).
  • The log is read by Spark itself. During the experiments the Spark UI showed queries named Scan ExistingRDD Delta Table State: Delta uses a small Spark job to load and filter the log.
Note. The commit-replay and checkpoint rules above are general Delta behaviour confirmed here by the real files; the formal protocol document could not be fetched while writing, so details such as multi-part and V2 checkpoints are not described.

FAQ

Questions people ask

What does the _delta_log folder contain?

Numbered JSON commit files that record add, remove, metaData, protocol and commitInfo actions, plus periodic checkpoint Parquet files and a _last_checkpoint pointer.

Where are the min and max values kept?

Twice: per file inside the add action in the delta log, and per column chunk inside each Parquet file's footer. Delta uses the first to skip files, and the Parquet reader uses the second to skip row groups.

Does removing a file delete it?

No. A remove action only marks the file as no longer part of the table. The file stays until VACUUM deletes files older than the retention period.

← Start here2 · Read path →