Home › Learn › Databricks interview prep › Skew and shuffle
Databricks interview prep · data engineering
Skew, salting and avoiding shuffle
What a shuffle is, how to spot skew, the fixes in order (AQE, hints, key isolation, salting with code), and how to shuffle less today.
Defaults, limits and preview status change often. Check the current Databricks docs before quoting a number.
Skew is a shuffle problem: equal keys must meet, and one key is too big.
First
What a shuffle is
When an operation needs all rows with the same key in one place (join, groupBy, distinct, window, repartition), Spark moves rows across the network. Map tasks write their output split by target partition; reduce tasks fetch their piece from every map task. It costs disk, network, serialization and sorting, so it is usually the most expensive step.
No shuffle: filter, select, column expressions, union, coalesce (decreasing), broadcast joins.
The problem
Recognising skew
- Spark UI stage page: the slowest task takes 5 times the median or more; shuffle read size per task is very uneven.
- One task spills to disk while others do not; one executor runs out of memory.
- A job sits at "199 of 200 tasks" for a long time.
- Typical causes: null or default keys (-1, unknown, empty string), one very popular customer or product, a dominant country, most data on one date.
Fix in this order
Fixes for skew
| Fix | How it works | Use when |
|---|---|---|
| Remove or isolate null and default keys | Filter before the join, process separately, union back. | The hot key is junk. Cheapest and most common. |
| AQE skew join | At run time, splits partitions much bigger than the median into several tasks. | Sort-merge joins. On by default; check it fired in the query profile. |
| Broadcast the small side | No shuffle of the big side. | One side is small. |
| Skew hint | Tell the optimizer the column and values. | AQE misses it and you know the hot keys. |
| Isolate the hot keys | Broadcast-join the hot rows, normal-join the rest, union. | A few known hot keys. |
| Salting | Spread a hot key over many partitions with a random suffix. | groupBy or join skew that AQE does not fix. |
| Two-stage aggregation | Partial aggregate, then final aggregate. | Skewed groupBy with sum, count, min, max. |
| Change the grain | Pre-aggregate upstream or pick a better key. | The design is the cause. |
| More partitions | Smaller partitions. | Only moderate skew. A single giant key still lands in one partition. |
Salting
How salting works
A hot key always hashes to the same partition. Add a random number so key K becomes K_0, K_1, … K_15, which hash to different partitions.
from pyspark.sql import functions as F
N = 16
# skewed groupBy: aggregate in two steps
salted = df.withColumn("salt", (F.rand() * N).cast("int"))
partial = salted.groupBy("key", "salt").agg(F.sum("amount").alias("p"))
final = partial.groupBy("key").agg(F.sum("p").alias("total"))
# skewed join: salt the big side, replicate the small side N times
big = big.withColumn("salt", (F.rand() * N).cast("int"))
small = small.withColumn("salt", F.explode(F.array([F.lit(i) for i in range(N)])))
joined = big.join(small, ["key", "salt"]).drop("salt")- The other side grows N times, so salt only the hot keys if you can: give normal keys salt 0 and do not replicate them.
- For an average, carry sum and count separately. Exact distinct count needs a different approach.
- rand() can differ on a task retry; derive the salt from a row id if results must be exact.
- Prefer AQE, hints and key isolation first. Salting is the fallback.
Today
Shuffling less
| Technique | Effect |
|---|---|
| Broadcast joins | No shuffle of the big table; AQE can switch to broadcast at run time. |
| Filter and select early | Fewer rows and columns in the shuffle. |
| Aggregate before joining | Reduces the big side when the join needs only aggregates. |
| Built-in aggregates | Spark pre-aggregates inside each map task before the shuffle. Avoid collecting lists. |
| AQE partition coalescing | No thousands of tiny shuffle tasks. |
| Liquid clustering and dynamic pruning | Read fewer files, so there is less to shuffle. |
| Avoid needless repartition() | Use coalesce to reduce partitions without a shuffle. |
| Reuse the same window partitioning | Each different partitionBy triggers a shuffle. |
| dropDuplicates on keys, approx_count_distinct | Cheaper than distinct on whole rows or exact counts. |
| Incremental processing | Shuffle only new or changed data. |
| Bucketing / storage-partitioned joins | Can avoid a shuffle but support on Delta is limited; rarely the answer. Check your runtime. |
The goal is not zero shuffles. It is less data, shuffled less often.
Routine
When a job is slow
- Find the slowest stage.
- Look at task time spread. Max far above median means skew; evenly slow means too much data or too little parallelism.
- Check shuffle size and spill.
- Count rows by the join or group key and look at the top 20 values.
- Apply the cheapest fix. Salting last.
- Compare stage times, not just total time.
Practice
Questions to rehearse
- Which operations cause a shuffle and which avoid one?
- How do you detect skew in the Spark UI?
- Explain salting for a join. What happens to the smaller table?
- How is AQE skew handling different from salting, and when does AQE not help?
- Why does raising shuffle partitions not fix one hot key?
- A groupBy on 10 TB has one task running for an hour. What do you do?
FAQ
Questions people ask
Does more shuffle partitions fix skew?
Only moderate skew. One key always lands in one partition.
Can salting break correctness?
For sums, counts, min and max it is safe with a two-step aggregation. Averages need sum and count carried separately.
Next
Where to go next
All Databricks interview topics · All data engineering · Databricks on this site