Broadcast Join vs Bucketing: Two Ways to Skip the Shuffle in Spark
A join shuffles both tables across the network. If one side is small you can broadcast it. If both are large, bucketing lets you do the shuffle once and never again.
Read MoreTag: Joins
Fix Spark Data Skew in Joins (One Task Runs Forever)
Your job is "99% done" for an hour because one task got all the data. That's skew. Here's how to fix it with AQE, broadcast joins, skew hints, and salting.
Read More