Spark Shuffle Explained: Why Wide Transformations Slow Your Job
A shuffle moves data across the network between executors. It is also the root cause of the three problems people spend most of their time debugging: skew, spill and out of memory errors.
Read More