Home › Learn › How reads and writes work
Delta Lake and Spark · from zero to advanced
How reads and writes work in Delta Lake and Spark
A picture-first guide, from zero to advanced, to what Spark and Delta Lake do when you read or write a table: files and the delta log, the read path, file layout, joins, shuffles, the driver, caching, Python and the JVM, Photon, query plans, the write paths, and why ClickHouse reads are fast.
When someone says a query is slow, or a write is expensive, the cause is nearly always one of a few mechanisms: which files were opened, how many bytes came off disk, whether data had to move between machines, or how much was rewritten. This section shows each mechanism with a drawing and then with a measurement, in the order a beginner needs them.
The whole story on one page. Each later page zooms into one box.
The path
Ten pages, in this order
- 1 · Files: What is on disk
A Delta table is a folder of Parquet files plus a transaction log. - 2 · Read path: How a read works
Follow a query from SQL to bytes. - 3 · Layout: Layout
Why the way files are organised decides how fast reads are: the small file problem measured, overlapping min and max ranges, columnar storage and compression with real chunk sizes, and OPTIMIZE, Z-order, liquid clustering and partitioning. - 4 · Joins: Joins
The join strategies Spark picks and why: broadcast hash, sort-merge, what AQE changes at run time, dynamic partition pruning that cut files read from 22 to 6, skew, and how organising files by the join key speeds joins up. - 5 · Inside Spark: Inside Spark
Narrow and wide transformations, the kinds of shuffle (hash, range, round-robin, single, broadcast), when work actually runs, what travels to the driver for count, take, show, collect and write, and what caching does to a select, with measured bytes. - 6 · Python and JVM: PySpark
How PySpark really works: the Python process, the Py4J bridge to the JVM, what the DataFrame API builds, what a Python UDF costs (measured), what collect converts, and why built-in functions are fast. - 7 · Photon: Photon
What the Photon engine replaces, why a vectorised C++ engine beats row-at-a-time JVM execution, which operators it covers, where it cannot help, and how to see it in a plan. - 8 · Plan and UI: Query plan, Spark UI and query profile
Read a real query plan bottom-up, then learn what the Spark UI tabs, the Databricks query profile, DESCRIBE DETAIL, DESCRIBE HISTORY and the delta log each tell you, with a symptom-to-tool table. - 9 · Write path: How writes work under the hood
What Delta does for each write: new files and a commit in the log. - 10 · ClickHouse: Why reads are so fast in ClickHouse
How ClickHouse reads data: sorted parts, one file per column, the sparse primary index with 8,192-row granules, skipping indexes, codecs, vectorised parallel execution, and how that compares with Delta and Parquet, from the ClickHouse docs.
How this was measured
Real runs, not invented numbers
Every number on these pages was produced by a script in the site repository and is measured on a laptop with Apache Spark 3.5.1 and open-source Delta Lake 3.2.0, local mode, 2 cores. Not Databricks: Photon, predictive I/O and liquid clustering were not available locally. Absolute times are tiny because the table is small. Use the ratios and counts (files read, bytes read, rows copied), which behave the same way on a terabyte.
- The table: 200,000 rows, 40 columns, 20 files of 10,000 rows, 11.3 MB on disk. A 1 TB table is the same picture with more files.
- Marked clearly: anything that only exists on Databricks (Photon, predictive I/O, liquid clustering, the query profile) comes from the Databricks documentation and was not measured.
- Honest gaps: the Delta protocol document could not be fetched during writing, so the log mechanics are shown from real log files, and the rules are labelled as general knowledge where they go beyond what was observed.
Words used
A small glossary
- Driver
- The one process that plans the query and talks to the log.
- Executor
- A worker process that runs tasks on slices of the data.
- Task
- One unit of work on one slice, usually one file or part of a file.
- Stage
- A group of tasks that can run without moving data between machines.
- Shuffle
- Moving rows between executors so that matching keys end up together.
- Parquet
- A column-oriented file format with min and max values in its footer.
- Delta log
- The folder of JSON commit files (plus checkpoints) that says which Parquet files make up the table.
- Data skipping
- Using min and max values to skip files that cannot match a filter.
- Predicate pushdown
- Handing a filter to the file reader so it can skip data early.
Related
Where it connects
Delta Lake internals simulator · Row vs column storage · Who maintains your tables · Photon and caches (compute guide) · Delta vs Iceberg
FAQ
Questions people ask
What is a Delta table, physically?
A folder of Parquet data files plus a _delta_log folder of JSON commit files and occasional checkpoints. The log says which files are currently part of the table and carries per-file statistics.
Why do reads touch the log first?
Because the log lists every live file with its minimum and maximum values. The driver uses that list to skip files before any data is read.
Are the numbers in this section from Databricks?
No. They come from open-source Delta Lake 3.2.0 on Spark 3.5.1 on a laptop. Databricks-only features are described from its documentation and marked as not measured.