Home › Learn › How reads and writes work › 5 · Inside Spark

Delta Lake and Spark · from zero to advanced

Inside Spark: narrow and wide, shuffles, the driver and caching

Narrow and wide transformations, the kinds of shuffle (hash, range, round-robin, single, broadcast), when work actually runs, what travels to the driver for count, take, show, collect and write, and what caching does to a select, with measured bytes.

Narrow and wide

The one idea that explains most Spark performance

Some steps work on each slice of data on its own (narrow). Others need rows from many slices to meet (wide) and so move data across the network. Narrow steps are cheap and chain together. Wide steps cost a shuffle and cut the job into stages.

Narrow: each output partition depends on one input partitionpartition 1filter + selectpartition 1partition 2filter + selectpartition 2partition 3filter + selectpartition 3Wide: output partitions need data from many inputsinput 1output 1input 2output 2input 3output 3No data moves between machines.Rows are sent across the network: a shuffle.Narrow steps are chained into one stage. Every wide step starts a new stage.

Lines that cross are the cost.

OperationKindExchangeShuffle written (measured)Tasks
filter + selectnarrownone0 B4
groupBy('country').sum()widehash838 B8
distinct()widehash656 B8
orderBy('amount')widerange32.6 MB10
repartition(8)wideround robin33.0 MB12
window partitionBy('country')widehash32.6 MB8
Measured. The surprise is in the bytes. groupBy and distinct shuffled only 838 B and 656 B, because Spark first aggregates inside each partition and shuffles only the partial results. orderBy, repartition and a window moved the whole dataset: about 32.6 MB, 33.0 MB and 32.6 MB, which is bigger than the 11.3 MB of compressed Parquet because shuffle data is not column-compressed that way.

Kinds of shuffle

Five kinds of exchange, one for each job

hashgroupBy, join, window,distinctsame key goes to the same partitionhashpartitioningrangeorderBysorted ranges per partitionrangepartitioninground robinrepartition(n)rows dealt out evenlyRoundRobinPartitioningsingleglobal aggregate,limiteverything to one partitionSinglePartitionbroadcastbroadcast joina copy to every executorBroadcastExchange

Each exchange line in a plan names how rows are redistributed.

  • Hash: rows with the same key go to the same partition. Used by group by, join and window.
  • Range: rows are split into sorted ranges. Used by order by. Spark first samples the data to pick the boundaries (general knowledge; the order by query ran an extra stage in the local test).
  • Round robin: rows are dealt out evenly, ignoring content. Used by repartition(n).
  • Single partition: everything goes to one place, so avoid it for big data.
  • Broadcast: a small table is copied to every executor.

When work actually happens

Nothing runs until an action

DataFrame steps like filter and select are transformations: they only extend a plan. Work starts when an action is called: count, collect, show, write. Then the driver turns the plan into stages at each exchange, and tasks run on executors.

CallKindWhat happens
spark.read…, .filter(), .select(), .join()transformationThe plan grows. No files are read. (A Delta read does load the log to know the schema.)
.explain()plan onlyPrints the plan without running it.
.count(), .collect(), .show(), .write…actionThe plan is optimised, split into stages and run.
.cache()markerNothing is stored until the first action runs.

What travels to the driver

Most actions send almost nothing back

Bytes the tasks sent back to the drivercount()13 KBtake(5)13 KBshow(20)12 KBgroupBy.agg().collect()21 KBwrite to a table (noop sink)13 KBcollect() of 2 columns2.2 MBcollect() of all 40 columns33.1 MBAlmost everything stays on the executors. Only collect-style actions ship the data itself.

Measured bytes returned to the driver by each action.

  • count(): each task sends one number. 13 KB in total, including the small log-reading tasks.
  • take(5) and show(): only as many partitions as needed are scanned (3 tasks), and a few rows come back.
  • write: the data stays on the executors and goes straight to storage; the driver only sees 13 KB of task results.
  • collect(): every row is sent to the driver. 200,000 rows of 2 columns was 2.2 MB (0.78 s); all 40 columns was 33.1 MB (1.97 s). At billions of rows this is the classic way to crash a driver.
  • A broadcast join also routes the small table through the driver before copying it out (general Spark behaviour, not measured). toPandas() is a collect and has the same limit; it was not measured because pandas is not installed in the test environment.
The safety limit. Spark caps the total result size with spark.driver.maxResultSize. I believe its default is 1 GB, but I did not re-check that against the docs.

Caching

What cache() does to a select

Parquet files11.3 MB on diskcache() / persist()first action fills itread onceExecutor memory17.3 MB heldLater actionsread memoryBefore the cache: 20 files read. After: 0 files read. Time 0.15 s → 0.12 s (tiny table).The cache holds every column it was given, decoded: 17.3 MB in memory against 11.3 MB of compressed Parquet.

Cache moves reads from files to memory, at a memory cost.

Measured. On the 11.3 MB table, a cached query read 0 files instead of 20 and ran in 0.12 s against 0.15 s. The table is tiny, so the win is small here; at scale the saving is the I/O. The cached data took 17.3 MB of memory for the whole table, more than the 11.3 MB of files, because it keeps every column decoded.
  • Select first, cache second. Cache only the columns and rows you will reuse; caching a 40-column DataFrame stores all 40.
  • Cache only what you reuse. If it is read once it is pure cost.
  • Delta also caches its own log state in small cached datasets, which showed up as extra entries in the storage list.
  • Databricks adds a separate disk cache for remote files on local SSD on supported compute. It is covered in the compute guide and was not measured here.

What to do

Rules that follow

DoVerdict
Chain narrow steps; count the exchanges in the planPursue
Reduce data before a wide step (filter, select, aggregate)Pursue
Use take, show or write instead of collect on big dataPursue
Collect a large table to the driverDrop
Cache a DataFrame you read onceDrop

FAQ

Questions people ask

What is the difference between narrow and wide transformations?

Narrow transformations such as filter and select work on one partition at a time and need no data movement. Wide transformations such as group by, join and order by need rows from many partitions and cause a shuffle, which starts a new stage.

Why did group by shuffle so little data?

Spark first aggregates within each partition and shuffles only the partial results. In the local test a group by moved 838 bytes while an order by of the same table moved 32.6 MB.

What gets sent to the driver?

Only what the action returns. count, take, show and write send kilobytes. collect sends every row, 33.1 MB for all 40 columns of 200,000 rows in the test, which is how drivers run out of memory on large tables.

← 4 · Joins6 · Python and JVM →