Home › Learn › How reads and writes work

Delta Lake and Spark · from zero to advanced

How reads and writes work in Delta Lake and Spark

A picture-first guide, from zero to advanced, to what Spark and Delta Lake do when you read or write a table: files and the delta log, the read path, file layout, joins, shuffles, the driver, caching, Python and the JVM, Photon, query plans, the write paths, and why ClickHouse reads are fast.

When someone says a query is slow, or a write is expensive, the cause is nearly always one of a few mechanisms: which files were opened, how many bytes came off disk, whether data had to move between machines, or how much was rewritten. This section shows each mechanism with a drawing and then with a measurement, in the order a beginner needs them.

Your querySQL or DataFrame(Python, SQL, Scala)Driverplans the queryreads the delta logpicks files to readExecutorstasks read files,filter, join, aggregateParquet files in storagecolumns, compressed,with min and max in the footer_delta_logwhich files, and their statsreadsReads: the log narrows the file list before any data is touched.Writes: executors write new Parquet files, then the driver adds a commit to the log.

The whole story on one page. Each later page zooms into one box.

The short version. A Delta table is Parquet files plus a log. A read asks the log which files could match, opens only those, and reads only the columns it needs. A write adds new files and then one line to the log. Everything else here explains how, why that sometimes goes wrong, and how to tell.

The path

Ten pages, in this order

  1. 1 · Files: What is on disk
    A Delta table is a folder of Parquet files plus a transaction log.
  2. 2 · Read path: How a read works
    Follow a query from SQL to bytes.
  3. 3 · Layout: Layout
    Why the way files are organised decides how fast reads are: the small file problem measured, overlapping min and max ranges, columnar storage and compression with real chunk sizes, and OPTIMIZE, Z-order, liquid clustering and partitioning.
  4. 4 · Joins: Joins
    The join strategies Spark picks and why: broadcast hash, sort-merge, what AQE changes at run time, dynamic partition pruning that cut files read from 22 to 6, skew, and how organising files by the join key speeds joins up.
  5. 5 · Inside Spark: Inside Spark
    Narrow and wide transformations, the kinds of shuffle (hash, range, round-robin, single, broadcast), when work actually runs, what travels to the driver for count, take, show, collect and write, and what caching does to a select, with measured bytes.
  6. 6 · Python and JVM: PySpark
    How PySpark really works: the Python process, the Py4J bridge to the JVM, what the DataFrame API builds, what a Python UDF costs (measured), what collect converts, and why built-in functions are fast.
  7. 7 · Photon: Photon
    What the Photon engine replaces, why a vectorised C++ engine beats row-at-a-time JVM execution, which operators it covers, where it cannot help, and how to see it in a plan.
  8. 8 · Plan and UI: Query plan, Spark UI and query profile
    Read a real query plan bottom-up, then learn what the Spark UI tabs, the Databricks query profile, DESCRIBE DETAIL, DESCRIBE HISTORY and the delta log each tell you, with a symptom-to-tool table.
  9. 9 · Write path: How writes work under the hood
    What Delta does for each write: new files and a commit in the log.
  10. 10 · ClickHouse: Why reads are so fast in ClickHouse
    How ClickHouse reads data: sorted parts, one file per column, the sparse primary index with 8,192-row granules, skipping indexes, codecs, vectorised parallel execution, and how that compares with Delta and Parquet, from the ClickHouse docs.

How this was measured

Real runs, not invented numbers

Every number on these pages was produced by a script in the site repository and is measured on a laptop with Apache Spark 3.5.1 and open-source Delta Lake 3.2.0, local mode, 2 cores. Not Databricks: Photon, predictive I/O and liquid clustering were not available locally. Absolute times are tiny because the table is small. Use the ratios and counts (files read, bytes read, rows copied), which behave the same way on a terabyte.

  • The table: 200,000 rows, 40 columns, 20 files of 10,000 rows, 11.3 MB on disk. A 1 TB table is the same picture with more files.
  • Marked clearly: anything that only exists on Databricks (Photon, predictive I/O, liquid clustering, the query profile) comes from the Databricks documentation and was not measured.
  • Honest gaps: the Delta protocol document could not be fetched during writing, so the log mechanics are shown from real log files, and the rules are labelled as general knowledge where they go beyond what was observed.

Words used

A small glossary

Driver
The one process that plans the query and talks to the log.
Executor
A worker process that runs tasks on slices of the data.
Task
One unit of work on one slice, usually one file or part of a file.
Stage
A group of tasks that can run without moving data between machines.
Shuffle
Moving rows between executors so that matching keys end up together.
Parquet
A column-oriented file format with min and max values in its footer.
Delta log
The folder of JSON commit files (plus checkpoints) that says which Parquet files make up the table.
Data skipping
Using min and max values to skip files that cannot match a filter.
Predicate pushdown
Handing a filter to the file reader so it can skip data early.

Related

Where it connects

Delta Lake internals simulator · Row vs column storage · Who maintains your tables · Photon and caches (compute guide) · Delta vs Iceberg

FAQ

Questions people ask

What is a Delta table, physically?

A folder of Parquet data files plus a _delta_log folder of JSON commit files and occasional checkpoints. The log says which files are currently part of the table and carries per-file statistics.

Why do reads touch the log first?

Because the log lists every live file with its minimum and maximum values. The driver uses that list to skip files before any data is read.

Are the numbers in this section from Databricks?

No. They come from open-source Delta Lake 3.2.0 on Spark 3.5.1 on a laptop. Databricks-only features are described from its documentation and marked as not measured.

1 · Files →