Home › Learn › How reads and writes work › 1 · Files
Delta Lake and Spark · from zero to advanced
What is on disk: Parquet files and the delta log
A Delta table is a folder of Parquet files plus a transaction log. See a real log (the add action with its min and max statistics, the commit files, a checkpoint) and the real inside of a Parquet file: row groups, column chunks, encodings and the footer.
From zero
A Delta table is a folder of files
There is no database server holding a Delta table. Open the folder and you see two kinds of things: Parquet data files, and a _delta_log folder that says which of those files currently make up the table.
The folder, drawn from a real table. The log is numbered; the checkpoint is a shortcut.
00000000000000000005.checkpoint.parquet and a _last_checkpoint file that says version 5, 10 live files, 16,530 bytes.Inside a data file
Parquet: columns stored in chunks, with a footer of statistics
Each data file is Parquet. Inside it, data is cut into row groups, and inside each row group every column is its own chunk. At the very end sits a footer with the schema, where each chunk starts, and the minimum and maximum of each column chunk.
Reading starts at the footer, not the top of the file.
event_id runs from 100000 to 109999. That one line is what lets a reader skip this whole file when you ask for event_id = 50.Inside the log
The add action: where the statistics live
Each commit file is plain JSON, one action per line. The important action is add: it names a data file and carries that file's statistics, so the driver can decide which files matter without opening any of them.
A real add action, lightly shortened. This is the line the driver reads.
The same commit also holds a protocol action (this table needs reader version 1, writer version 2), a metaData action (schema, partition columns, table settings), and a commitInfo action recording who did what: operation WRITE, mode Overwrite, isolation level Serializable, written by Apache-Spark/3.5.1 Delta-Lake/3.2.0.
Putting commits together
How a reader works out which files are live
Add and remove, replayed in order, give the table's current state.
- Every add makes a file live. Every remove makes it not live. The current table is what is left after replaying the commits in order.
- To avoid replaying thousands of small files, Delta writes a checkpoint every few commits: one Parquet file holding the whole live set. A reader loads the latest checkpoint and then only the JSON commits after it.
- Remove does not delete. The old Parquet file stays in the folder. That is why
VERSION AS OF 0still works after an overwrite, and why storage grows untilVACUUMruns (see the write path). - The log is read by Spark itself. During the experiments the Spark UI showed queries named
Scan ExistingRDD Delta Table State: Delta uses a small Spark job to load and filter the log.
FAQ
Questions people ask
What does the _delta_log folder contain?
Numbered JSON commit files that record add, remove, metaData, protocol and commitInfo actions, plus periodic checkpoint Parquet files and a _last_checkpoint pointer.
Where are the min and max values kept?
Twice: per file inside the add action in the delta log, and per column chunk inside each Parquet file's footer. Delta uses the first to skip files, and the Parquet reader uses the second to skip row groups.
Does removing a file delete it?
No. A remove action only marks the file as no longer part of the table. The file stays until VACUUM deletes files older than the retention period.