Home › Learn › How reads and writes work › 5 · Inside Spark
Delta Lake and Spark · from zero to advanced
Inside Spark: narrow and wide, shuffles, the driver and caching
Narrow and wide transformations, the kinds of shuffle (hash, range, round-robin, single, broadcast), when work actually runs, what travels to the driver for count, take, show, collect and write, and what caching does to a select, with measured bytes.
Narrow and wide
The one idea that explains most Spark performance
Some steps work on each slice of data on its own (narrow). Others need rows from many slices to meet (wide) and so move data across the network. Narrow steps are cheap and chain together. Wide steps cost a shuffle and cut the job into stages.
Lines that cross are the cost.
| Operation | Kind | Exchange | Shuffle written (measured) | Tasks |
|---|---|---|---|---|
filter + select | narrow | none | 0 B | 4 |
groupBy('country').sum() | wide | hash | 838 B | 8 |
distinct() | wide | hash | 656 B | 8 |
orderBy('amount') | wide | range | 32.6 MB | 10 |
repartition(8) | wide | round robin | 33.0 MB | 12 |
window partitionBy('country') | wide | hash | 32.6 MB | 8 |
groupBy and distinct shuffled only 838 B and 656 B, because Spark first aggregates inside each partition and shuffles only the partial results. orderBy, repartition and a window moved the whole dataset: about 32.6 MB, 33.0 MB and 32.6 MB, which is bigger than the 11.3 MB of compressed Parquet because shuffle data is not column-compressed that way.Kinds of shuffle
Five kinds of exchange, one for each job
Each exchange line in a plan names how rows are redistributed.
- Hash: rows with the same key go to the same partition. Used by group by, join and window.
- Range: rows are split into sorted ranges. Used by order by. Spark first samples the data to pick the boundaries (general knowledge; the order by query ran an extra stage in the local test).
- Round robin: rows are dealt out evenly, ignoring content. Used by repartition(n).
- Single partition: everything goes to one place, so avoid it for big data.
- Broadcast: a small table is copied to every executor.
When work actually happens
Nothing runs until an action
DataFrame steps like filter and select are transformations: they only extend a plan. Work starts when an action is called: count, collect, show, write. Then the driver turns the plan into stages at each exchange, and tasks run on executors.
| Call | Kind | What happens |
|---|---|---|
spark.read…, .filter(), .select(), .join() | transformation | The plan grows. No files are read. (A Delta read does load the log to know the schema.) |
.explain() | plan only | Prints the plan without running it. |
.count(), .collect(), .show(), .write… | action | The plan is optimised, split into stages and run. |
.cache() | marker | Nothing is stored until the first action runs. |
What travels to the driver
Most actions send almost nothing back
Measured bytes returned to the driver by each action.
- count(): each task sends one number. 13 KB in total, including the small log-reading tasks.
- take(5) and show(): only as many partitions as needed are scanned (3 tasks), and a few rows come back.
- write: the data stays on the executors and goes straight to storage; the driver only sees 13 KB of task results.
- collect(): every row is sent to the driver. 200,000 rows of 2 columns was 2.2 MB (0.78 s); all 40 columns was 33.1 MB (1.97 s). At billions of rows this is the classic way to crash a driver.
- A broadcast join also routes the small table through the driver before copying it out (general Spark behaviour, not measured).
toPandas()is a collect and has the same limit; it was not measured because pandas is not installed in the test environment.
spark.driver.maxResultSize. I believe its default is 1 GB, but I did not re-check that against the docs.Caching
What cache() does to a select
Cache moves reads from files to memory, at a memory cost.
- Select first, cache second. Cache only the columns and rows you will reuse; caching a 40-column DataFrame stores all 40.
- Cache only what you reuse. If it is read once it is pure cost.
- Delta also caches its own log state in small cached datasets, which showed up as extra entries in the storage list.
- Databricks adds a separate disk cache for remote files on local SSD on supported compute. It is covered in the compute guide and was not measured here.
What to do
Rules that follow
| Do | Verdict |
|---|---|
| Chain narrow steps; count the exchanges in the plan | Pursue |
| Reduce data before a wide step (filter, select, aggregate) | Pursue |
| Use take, show or write instead of collect on big data | Pursue |
| Collect a large table to the driver | Drop |
| Cache a DataFrame you read once | Drop |
FAQ
Questions people ask
What is the difference between narrow and wide transformations?
Narrow transformations such as filter and select work on one partition at a time and need no data movement. Wide transformations such as group by, join and order by need rows from many partitions and cause a shuffle, which starts a new stage.
Why did group by shuffle so little data?
Spark first aggregates within each partition and shuffles only the partial results. In the local test a group by moved 838 bytes while an order by of the same table moved 32.6 MB.
What gets sent to the driver?
Only what the action returns. count, take, show and write send kilobytes. collect sends every row, 33.1 MB for all 40 columns of 200,000 rows in the test, which is how drivers run out of memory on large tables.