Databricks Troubleshooting

Fix Spark Data Skew in Joins (One Task Runs Forever)

Your job is "99% done" for an hour because one task got all the data. That's skew. Here's how to fix it with AQE, broadcast joins, skew hints, and salting.

Databricks Troubleshooting

You know the feeling. A Spark job flies through 199 of 200 tasks in seconds, then sits at “99% complete” for an hour on the last one. No error, no crash — just one task grinding away, maybe spilling to disk or eventually running the executor out of memory. That single straggler is almost always data skew.

⚡ Quick fix

  • Make sure Adaptive Query Execution (AQE) skew handling is on — it splits skewed partitions automatically and is the first line of defense.
  • If one side of the join is small, broadcast it — a broadcast join has no shuffle, so skew can’t bite.
  • For a known hot key, add a skew hint (Databricks) so the optimizer plans for it.
  • For extreme skew, salt the key to spread the hot value across many partitions.
spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true")

What data skew actually is

To join two tables, Spark shuffles rows so that all rows with the same key land in the same partition, on the same executor. That works beautifully when keys are spread evenly. It falls apart when one key value is wildly more common than the rest.

Say you join orders to customers on customer_id, and one giant customer (or a junk value like customer_id = 0 or null) accounts for 40% of all orders. Every one of those rows is shuffled into a single partition. One task now has to process 40% of the data alone while the others sit idle. That’s your hour-long straggler.

One hot key sends most of the data to a single partition — the straggler that holds up the whole job.

Confirm it’s skew (Spark UI)

Don’t guess — check. In the Spark UI, open the slow stage and look at the task metrics. Skew is unmistakable: the max task duration and shuffle-read size are many times larger than the median. A few seconds median, twenty minutes max, is textbook skew.

To find the culprit key, count rows per key and look at the top of the list:

from pyspark.sql.functions import col

orders.groupBy("customer_id").count().orderBy(col("count").desc()).show(10)
# One value with a wildly higher count than the rest = your hot key

The fix, from easiest to heaviest

1. Let AQE handle it

Modern Spark and Databricks have Adaptive Query Execution, which detects a skewed partition at runtime and splits it into smaller sub-partitions across more tasks. It’s on by default on recent runtimes — confirm it, and many skew problems just disappear:

spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true")

2. Broadcast the small side

If one table is small (think dimension tables — customers, products), broadcast it to every executor. A broadcast join skips the shuffle entirely, so there’s no partition for the hot key to overload:

from pyspark.sql.functions import broadcast

result = orders.join(broadcast(customers), "customer_id")

3. Add a skew hint (Databricks)

If you already know which key is skewed, tell the engine directly so it plans around it:

-- SQL
SELECT /*+ SKEW('orders', 'customer_id') */ *
FROM orders JOIN customers USING (customer_id);

-- DataFrame
result = orders.hint("skew", "customer_id").join(customers, "customer_id")

4. Salt the key (for extreme skew)

When neither broadcast nor AQE is enough, salting spreads the hot key across many partitions by appending a random number, joining on the salted key, and dropping the salt after. The big side gets a random salt; the small side is duplicated across every salt value so matches still find each other:

from pyspark.sql.functions import col, concat, lit, floor, rand

N = 16  # number of salt buckets

# Big/skewed side: one random salt per row
orders_salted = (orders
    .withColumn("salt", floor(rand() * N))
    .withColumn("k", concat(col("customer_id"), lit("_"), col("salt"))))

# Small side: replicate every row across all salt buckets
salts = spark.range(N).withColumnRenamed("id", "salt")
customers_exploded = (customers.crossJoin(salts)
    .withColumn("k", concat(col("customer_id"), lit("_"), col("salt"))))

result = orders_salted.join(customers_exploded, "k").drop("salt", "k")

Salting is powerful but adds complexity, so reach for it only after AQE, broadcast, and hints haven’t solved it.

How to verify it’s fixed

  • In the Spark UI, the stage’s max task duration is now close to the median — no lone straggler.
  • The job finishes in a fraction of the time, with no disk spill or executor OOM on that stage.
  • Results are identical to before (salting especially — always confirm the row count matches).

How to prevent it

  • Keep AQE enabled — it quietly handles most skew for free.
  • Filter out junk hot keys like null or 0 before the join if they’re not real data.
  • Broadcast small dimensions as a default pattern to avoid shuffles altogether.
  • Know your data’s distribution — a quick group-by-count during development catches skew before production does.

When it’s actually something else

  • Under-sized cluster: if all tasks are slow (not one), the data may simply be too big for the cluster — that’s sizing, not skew.
  • Too few or too many partitions: uniformly slow shuffles can be a partition-count problem, tuned via spark.sql.shuffle.partitions or AQE coalescing.
  • Spill from wide rows: very large individual rows can spill without classic key skew.

Frequently Asked Questions

What is data skew in Spark?

Data skew is an uneven distribution of data across partitions, usually caused by one key value being far more common than others. During a join or aggregation, all rows with that key land in a single partition, so one task processes most of the data while the rest sit idle, creating a slow straggler.

How do I fix a Spark join stuck at 99%?

A join stuck on the last task is usually skew. Confirm Adaptive Query Execution skew handling is enabled, broadcast the smaller table if it fits, add a skew hint for a known hot key, and for extreme cases salt the key to spread it across partitions.

What is salting in Spark?

Salting is a technique to fix severe skew by appending a random number to the join key so a single hot value is spread across many partitions. The larger side gets a random salt per row, the smaller side is replicated across all salt values, and the salt is dropped after the join.

Does AQE fix data skew automatically?

Adaptive Query Execution handles many skew cases automatically by detecting skewed partitions at runtime and splitting them across more tasks. It is enabled by default on recent Spark and Databricks runtimes and resolves a large share of skew without code changes, though extreme skew may still need broadcasting, hints, or salting.

How do I know if my Spark job has skew?

In the Spark UI, open the slow stage and compare task metrics: with skew, the maximum task duration and shuffle-read size are far larger than the median. You can also group by the join key and count rows — a single value with a much higher count than the rest is your skewed key.

Related Posts: