Home › Learn › How reads and writes work › 9 · Write path
Delta Lake and Spark · from zero to advanced
How writes work under the hood: append, overwrite, update, delete, merge
What Delta does for each write: new files and a commit in the log. Real logs and counters for append, update, delete, delete with deletion vectors, merge, optimize and overwrite, why rewriting 10,000 rows to change 11 happens, and how time travel keeps old files.
The sequence
Every write is: write new files, then commit
Files first, then one atomic commit.
- Data files are never edited. A Parquet file, once written, does not change. A write adds new files, and the log says which files are live.
- The commit is atomic. The log entry for version N+1 either exists or does not, so a reader never sees half a write. If a job dies before step 4, it leaves unused files and no change to the table.
- The log records how it was done. Every commit in the experiments logged
readVersion(the version the writer started from),isolationLevel(Serializable) andisBlindAppend(true only if the write read nothing first). - Concurrency (general knowledge, not measured here): two writers can run at once. When a second tries to commit a version someone has already taken, Delta checks whether the changes touch the same files and either retries on top of the winner or fails with a conflict. The Delta protocol document could not be fetched to confirm the details.
All the write paths
What each one does to the files
From real commits. Counts of files added and removed are exact.
| Write | Operation in the log | add actions | remove actions | Blind append | Isolation |
|---|---|---|---|---|---|
| INSERT (append) | WRITE | 2 | 0 | yes | Serializable |
| UPDATE, one file | UPDATE | 1 | 1 | no | Serializable |
| UPDATE, every file | UPDATE | 2 | 10 | no | Serializable |
| DELETE, rewrite | DELETE | 1 | 1 | no | Serializable |
| DELETE, deletion vector | DELETE | 1 | 1 | no | Serializable |
| MERGE, one file | MERGE | 1 | 1 | no | Serializable |
| MERGE, every file | MERGE | 2 | 10 | no | Serializable |
| OPTIMIZE | OPTIMIZE | 1 | 20 | no | SnapshotIsolation |
| OVERWRITE | WRITE | 2 | 10 | no | Serializable |
Method. Each operation ran on its own fresh 10-file table of 100,000 rows, so the counts are not muddied by earlier writes.
Append and insert
The cheap one
An INSERT of 5,000 rows added 2 new files and removed none; the commit has only add actions (0.42 s). No existing file is read or touched, which is why appends are the cheapest write, and why concurrent appends usually do not conflict (general knowledge, not measured here). The downside is the small file problem: many small appends make many small files, which OPTIMIZE later fixes.
Update
Changing 11 rows rewrote 10,000
Parquet files cannot be edited in place, so an UPDATE finds the files that contain matching rows, reads them whole, writes new copies with the changes applied, and removes the old ones from the log. This is called copy on write.
Rows rewritten to change a few rows. Spread-out changes are the worst case.
- Layout decides the cost. If rows that change together sit in the same files, an update rewrites few files. Clustering by the update key helps.
- A rewrite also reshapes the table: the second update above turned 10 files into 2, because the rewrite produced new files sized by the write, not the original layout.
Delete
Rewrite, or mark with a deletion vector
Without deletion vectors, a delete is an update that drops rows: the whole file is rewritten without them. With them, nothing is rewritten.
A deletion vector turns a delete into a metadata change.
delta.enableDeletionVectors = true it added 1 deletion vector, rewrote 0 files and copied 0 rows. The log entry shows a deletionVector with storageType u, size 31 bytes and cardinality 50. Reading the table afterwards returned 99,950 of the original 100,000 rows.- The trade: deletes become cheap, and later reads do a little extra work until OPTIMIZE or a rewrite removes the marked rows (the usual reason for running OPTIMIZE after many deletes; general knowledge).
- In the log the file shows as a remove and an add of the same file, the add now carrying the vector. No data file was written.
Merge
Two phases: find the touched files, then rewrite them
MERGE joins a source to the target, then applies updates, deletes and inserts. Internally it first finds which target files contain matches, then rewrites only those files and writes any inserted rows as new files.
- The cost of a merge is the files it touches, not the rows it changes. Put a filter on the merge condition (for example a date) so the search covers fewer files.
- Not measured: how the source and target are joined inside MERGE, and Databricks features that speed it up (the docs say dynamic file pruning in MERGE needs Photon).
Compaction
OPTIMIZE rewrites many files as one
After five tiny appends the table had 20 files. OPTIMIZE removed 20 and added 1. The commit marks these as not changing data, so streams and readers can ignore it.
Overwrite and time travel
Old files stay until VACUUM
An overwrite removed all 10 files from the live set and added 2 new ones. The 100,000-row version is still there: VERSION AS OF 0 returned 100,000 rows while the current table had 20,000. The folder held 12 Parquet files but only 2 were live.
- Time travel is just reading an older log version. No data was copied for it.
- Storage keeps growing with every rewrite until
VACUUMdeletes files older than the retention period (the Databricks docs namedelta.deletedFileRetentionDuration, default 7 days). VACUUM was not run in these tests.
What to do
Write rules that follow
| Do | Because | Verdict |
|---|---|---|
| Batch appends and compact | Many tiny commits make small files | Pursue |
| Cluster data by the keys you update or merge on | Fewer files are touched | Pursue |
| Turn on deletion vectors for delete-heavy tables | A delete stops rewriting files | Pursue |
| Filter the MERGE condition | Fewer target files to search | Pursue |
| Update scattered rows one at a time | Each update rewrites whole files | Drop |
| Assume an overwrite frees disk | Old files remain until VACUUM | Drop |
FAQ
Questions people ask
What does Delta do when you update a row?
It finds the files holding matching rows, reads them whole, writes new copies with the change applied, and records a remove for the old file and an add for the new one. In the local test, updating 11 rows rewrote 10,000.
What are deletion vectors?
A small side file that marks deleted row positions, so a delete does not rewrite the data file. In the test, deleting 50 rows with deletion vectors rewrote 0 files and copied 0 rows, against 1 file and 9,950 rows copied without them.
Why does storage not shrink after an overwrite?
Delta only marks the old files as removed in the log. They stay on disk so older versions can be read, until VACUUM deletes files past the retention period.