Home › Learn › How reads and writes work › 9 · Write path

Delta Lake and Spark · from zero to advanced

How writes work under the hood: append, overwrite, update, delete, merge

What Delta does for each write: new files and a commit in the log. Real logs and counters for append, update, delete, delete with deletion vectors, merge, optimize and overwrite, why rewriting 10,000 rows to change 11 happens, and how time travel keeps old files.

The sequence

Every write is: write new files, then commit

1 Planread the logat version N2 Write filesexecutors writenew Parquet files;nobody can seethem yet3 Gatherdriver collectseach new file'spath, size, stats4 Commitdriver createsN+1.json in the log,atomically5 Visiblereaders now seeversion N+1If another writer committed N+1 first: check whether the twochanges conflict, then retry on top of it or failThe commit is the only step that matters:before it, a failed write leaves only unused files.

Files first, then one atomic commit.

  • Data files are never edited. A Parquet file, once written, does not change. A write adds new files, and the log says which files are live.
  • The commit is atomic. The log entry for version N+1 either exists or does not, so a reader never sees half a write. If a job dies before step 4, it leaves unused files and no change to the table.
  • The log records how it was done. Every commit in the experiments logged readVersion (the version the writer started from), isolationLevel (Serializable) and isBlindAppend (true only if the write read nothing first).
  • Concurrency (general knowledge, not measured here): two writers can run at once. When a second tries to commit a version someone has already taken, Delta checks whether the changes touch the same files and either retries on top of the winner or fails with a conflict. The Delta protocol document could not be fetched to confirm the details.

All the write paths

What each one does to the files

each square is a data file: white = kept red = removed green = added yellow = same file plus a deletion vectorINSERT / append0 removed, 2 addedUPDATE, rows in one file1 removed, 1 addedUPDATE, rows in every file10 removed, 2 addedDELETE (rewrite)1 removed, 1 addedDELETE (deletion vector)0 rewritten, 1 deletion vectorMERGE, keys in one file1 removed, 1 addedMERGE, keys in every file10 removed, 2 addedOPTIMIZE (20 small files)20 removed, 1 addedOVERWRITE10 removed, 2 added

From real commits. Counts of files added and removed are exact.

WriteOperation in the logadd actionsremove actionsBlind appendIsolation
INSERT (append)WRITE20yesSerializable
UPDATE, one fileUPDATE11noSerializable
UPDATE, every fileUPDATE210noSerializable
DELETE, rewriteDELETE11noSerializable
DELETE, deletion vectorDELETE11noSerializable
MERGE, one fileMERGE11noSerializable
MERGE, every fileMERGE210noSerializable
OPTIMIZEOPTIMIZE120noSnapshotIsolation
OVERWRITEWRITE210noSerializable

Method. Each operation ran on its own fresh 10-file table of 100,000 rows, so the counts are not muddied by earlier writes.

Append and insert

The cheap one

An INSERT of 5,000 rows added 2 new files and removed none; the commit has only add actions (0.42 s). No existing file is read or touched, which is why appends are the cheapest write, and why concurrent appends usually do not conflict (general knowledge, not measured here). The downside is the small file problem: many small appends make many small files, which OPTIMIZE later fixes.

Update

Changing 11 rows rewrote 10,000

Parquet files cannot be edited in place, so an UPDATE finds the files that contain matching rows, reads them whole, writes new copies with the changes applied, and removes the old ones from the log. This is called copy on write.

UPDATE, 11 rows in one file10,000 rows rewritten to change 11UPDATE, 1,042 rows in every file100,000 rows rewritten to change 1,042DELETE, 50 rows (rewrite)10,000 rows rewritten to change 50DELETE, 50 rows (deletion vector)0 rows rewritten to change 50MERGE, 1,000 updates in one file10,000 rows rewritten to change 1,000MERGE, 100 updates, one per file100,000 rows rewritten to change 100

Rows rewritten to change a few rows. Spread-out changes are the worst case.

Measured. Updating 11 rows that lived in one file removed 1 file and added 1, with 11 rows updated and 9,989 copied unchanged. Updating about 1,000 rows spread through every file removed 10 files and added 2, copying 98,958 rows. Where the changed rows live matters far more than how many there are.
  • Layout decides the cost. If rows that change together sit in the same files, an update rewrites few files. Clustering by the update key helps.
  • A rewrite also reshapes the table: the second update above turned 10 files into 2, because the rewrite produced new files sized by the write, not the original layout.

Delete

Rewrite, or mark with a deletion vector

Without deletion vectors, a delete is an update that drops rows: the whole file is rewritten without them. With them, nothing is rewritten.

Data file (Parquet)10,000 rows, untouchedstill the same bytes on diskDeletion vectortiny: 31 bytes heremarks 50 row positionspointed to by the add actionA reader opens the fileand skips the marked rowsDelete becomes a metadata change. The cost moves to later: reads must apply the vector until OPTIMIZE or a rewrite purges it.

A deletion vector turns a delete into a metadata change.

Measured. Deleting 50 rows: by rewrite it removed 1 file, added 1 and copied 9,950 rows. With delta.enableDeletionVectors = true it added 1 deletion vector, rewrote 0 files and copied 0 rows. The log entry shows a deletionVector with storageType u, size 31 bytes and cardinality 50. Reading the table afterwards returned 99,950 of the original 100,000 rows.
  • The trade: deletes become cheap, and later reads do a little extra work until OPTIMIZE or a rewrite removes the marked rows (the usual reason for running OPTIMIZE after many deletes; general knowledge).
  • In the log the file shows as a remove and an add of the same file, the add now carrying the vector. No data file was written.

Merge

Two phases: find the touched files, then rewrite them

MERGE joins a source to the target, then applies updates, deletes and inserts. Internally it first finds which target files contain matches, then rewrites only those files and writes any inserted rows as new files.

Measured. Merging 1,000 matching keys that sat in one file (plus 1,000 new keys) removed 1 file and added 1: 1,000 updated, 1,000 inserted, 9,000 copied. Merging only 100 keys, but one in each of the 10 files, removed 10 files and rewrote 100,000 rows to change 100.
  • The cost of a merge is the files it touches, not the rows it changes. Put a filter on the merge condition (for example a date) so the search covers fewer files.
  • Not measured: how the source and target are joined inside MERGE, and Databricks features that speed it up (the docs say dynamic file pruning in MERGE needs Photon).

Compaction

OPTIMIZE rewrites many files as one

After five tiny appends the table had 20 files. OPTIMIZE removed 20 and added 1. The commit marks these as not changing data, so streams and readers can ignore it.

Overwrite and time travel

Old files stay until VACUUM

An overwrite removed all 10 files from the live set and added 2 new ones. The 100,000-row version is still there: VERSION AS OF 0 returned 100,000 rows while the current table had 20,000. The folder held 12 Parquet files but only 2 were live.

  • Time travel is just reading an older log version. No data was copied for it.
  • Storage keeps growing with every rewrite until VACUUM deletes files older than the retention period (the Databricks docs name delta.deletedFileRetentionDuration, default 7 days). VACUUM was not run in these tests.

What to do

Write rules that follow

DoBecauseVerdict
Batch appends and compactMany tiny commits make small filesPursue
Cluster data by the keys you update or merge onFewer files are touchedPursue
Turn on deletion vectors for delete-heavy tablesA delete stops rewriting filesPursue
Filter the MERGE conditionFewer target files to searchPursue
Update scattered rows one at a timeEach update rewrites whole filesDrop
Assume an overwrite frees diskOld files remain until VACUUMDrop

FAQ

Questions people ask

What does Delta do when you update a row?

It finds the files holding matching rows, reads them whole, writes new copies with the change applied, and records a remove for the old file and an add for the new one. In the local test, updating 11 rows rewrote 10,000.

What are deletion vectors?

A small side file that marks deleted row positions, so a delete does not rewrite the data file. In the test, deleting 50 rows with deletion vectors rewrote 0 files and copied 0 rows, against 1 file and 9,950 rows copied without them.

Why does storage not shrink after an overwrite?

Delta only marks the old files as removed in the log. They stay on disk so older versions can be read, until VACUUM deletes files past the retention period.

← 8 · Plan and UI10 · ClickHouse →