Broadcast Join vs Bucketing: Two Ways to Skip the Shuffle in Spark
A join shuffles both tables across the network. If one side is small you can broadcast it. If both are large, bucketing lets you do the shuffle once and never again.
Read MoreTag: Broadcast Join
Spark Driver OOM: Why collect() and Broadcast Joins Crash Your Job
Your executors have 64 GB and the job still dies with OutOfMemoryError. The culprit is the one part of Spark everyone forgets about — the Driver. Here's why it crashes and how to size it, in plain English.
Read More