AQE re-plans your query while it runs, using real numbers instead of estimates. Here are its three optimisations, every setting that controls them, and the default that quietly ignores your tuning.
Read MoreExecutor out of memory means one task was handed more data than its share of memory could hold. Here is how to tell it apart from driver OOM, what the two different errors mean, and the fixes in order.
Read MoreA join shuffles both tables across the network. If one side is small you can broadcast it. If both are large, bucketing lets you do the shuffle once and never again.
Read MoreSpill means Spark ran out of memory in the middle of a task and dumped its working data to disk. Your job does not fail — it just gets slower, with nothing in the logs to explain why.
Read MoreA shuffle moves data across the network between executors. It is also the root cause of the three problems people spend most of their time debugging: skew, spill and out of memory errors.
Read More41,600 files holding 14.8 GB took 34 minutes just to scan. Here is what causes the small files problem, the exact numbers that reveal it in the Spark UI, and four ways to fix it.
Read MoreSpark runs on the JVM. Python UDFs do not. Every row has to be converted between the two — and here is what that costs, how to spot it, and how to fix it.
Read MoreSix redundancy options are really three yes/no questions. What each one protects against, why none of them is a backup, and how to pick per medallion layer.
Read More
