Home › Learn › Databricks interview prep › Skew and shuffle

Databricks interview prep · data engineering

Skew, salting and avoiding shuffle

What a shuffle is, how to spot skew, the fixes in order (AQE, hints, key isolation, salting with code), and how to shuffle less today.

Maintenance commandsRow-level securityCompute choiceJoinsSkew and shuffleAQE, Photon, POCost and performance leversDeclarative and semanticAsset BundlesLakeflow ConnectUniFormVector Searchdbt and ODCSEgress and ingressMaintenance in picturesClustering and compactionMaintenance strategySlow writesBackup and recoveryTaggingQuestion bank

Defaults, limits and preview status change often. Check the current Databricks docs before quoting a number.

Rows split by keyOne key is hugeOne task gets most of thedataThe stage waits for it

Skew is a shuffle problem: equal keys must meet, and one key is too big.

First

What a shuffle is

When an operation needs all rows with the same key in one place (join, groupBy, distinct, window, repartition), Spark moves rows across the network. Map tasks write their output split by target partition; reduce tasks fetch their piece from every map task. It costs disk, network, serialization and sorting, so it is usually the most expensive step.

No shuffle: filter, select, column expressions, union, coalesce (decreasing), broadcast joins.

The problem

Recognising skew

Fix in this order

Fixes for skew

FixHow it worksUse when
Remove or isolate null and default keysFilter before the join, process separately, union back.The hot key is junk. Cheapest and most common.
AQE skew joinAt run time, splits partitions much bigger than the median into several tasks.Sort-merge joins. On by default; check it fired in the query profile.
Broadcast the small sideNo shuffle of the big side.One side is small.
Skew hintTell the optimizer the column and values.AQE misses it and you know the hot keys.
Isolate the hot keysBroadcast-join the hot rows, normal-join the rest, union.A few known hot keys.
SaltingSpread a hot key over many partitions with a random suffix.groupBy or join skew that AQE does not fix.
Two-stage aggregationPartial aggregate, then final aggregate.Skewed groupBy with sum, count, min, max.
Change the grainPre-aggregate upstream or pick a better key.The design is the cause.
More partitionsSmaller partitions.Only moderate skew. A single giant key still lands in one partition.

Salting

How salting works

A hot key always hashes to the same partition. Add a random number so key K becomes K_0, K_1, … K_15, which hash to different partitions.

from pyspark.sql import functions as F
N = 16

# skewed groupBy: aggregate in two steps
salted  = df.withColumn("salt", (F.rand() * N).cast("int"))
partial = salted.groupBy("key", "salt").agg(F.sum("amount").alias("p"))
final   = partial.groupBy("key").agg(F.sum("p").alias("total"))

# skewed join: salt the big side, replicate the small side N times
big   = big.withColumn("salt", (F.rand() * N).cast("int"))
small = small.withColumn("salt", F.explode(F.array([F.lit(i) for i in range(N)])))
joined = big.join(small, ["key", "salt"]).drop("salt")

Today

Shuffling less

TechniqueEffect
Broadcast joinsNo shuffle of the big table; AQE can switch to broadcast at run time.
Filter and select earlyFewer rows and columns in the shuffle.
Aggregate before joiningReduces the big side when the join needs only aggregates.
Built-in aggregatesSpark pre-aggregates inside each map task before the shuffle. Avoid collecting lists.
AQE partition coalescingNo thousands of tiny shuffle tasks.
Liquid clustering and dynamic pruningRead fewer files, so there is less to shuffle.
Avoid needless repartition()Use coalesce to reduce partitions without a shuffle.
Reuse the same window partitioningEach different partitionBy triggers a shuffle.
dropDuplicates on keys, approx_count_distinctCheaper than distinct on whole rows or exact counts.
Incremental processingShuffle only new or changed data.
Bucketing / storage-partitioned joinsCan avoid a shuffle but support on Delta is limited; rarely the answer. Check your runtime.

The goal is not zero shuffles. It is less data, shuffled less often.

Routine

When a job is slow

  1. Find the slowest stage.
  2. Look at task time spread. Max far above median means skew; evenly slow means too much data or too little parallelism.
  3. Check shuffle size and spill.
  4. Count rows by the join or group key and look at the top 20 values.
  5. Apply the cheapest fix. Salting last.
  6. Compare stage times, not just total time.

Practice

Questions to rehearse

  1. Which operations cause a shuffle and which avoid one?
  2. How do you detect skew in the Spark UI?
  3. Explain salting for a join. What happens to the smaller table?
  4. How is AQE skew handling different from salting, and when does AQE not help?
  5. Why does raising shuffle partitions not fix one hot key?
  6. A groupBy on 10 TB has one task running for an hour. What do you do?
In an interview, say: A shuffle moves rows so equal keys meet; skew is when one key is so big that one task does most of the work. I try the cheap fixes first: drop junk keys, broadcast, let AQE split, filter and aggregate early. Then key isolation, and salting last.

FAQ

Questions people ask

Does more shuffle partitions fix skew?

Only moderate skew. One key always lands in one partition.

Can salting break correctness?

For sums, counts, min and max it is safe with a two-step aggregation. Averages need sum and count carried separately.

Next

Where to go next

All Databricks interview topics · All data engineering · Databricks on this site