PySpark withColumn vs withColumns vs select: Which to Use
Three ways to add or change columns in PySpark — and one of them, used in a loop, quietly wrecks performance. Here's what each does and when to reach for it.
Read MoreTag: Spark Optimization
Full Recompute vs Incremental Processing: Which to Use When
Everyone assumes incremental is always the better choice. It isn't — for small tables it's over-engineering. Here's the real trade-off, a simple decision tree, and how to detect changes.
Read More
MERGE vs INSERT OVERWRITE vs APPEND: How to Write a Delta Table
Reaching for MERGE by default is one of the most expensive habits in data engineering. The right write pattern depends on your source — here's the decision, one question at a time.
Read More
Liquid Clustering vs Z-ORDER vs Partitioning: Which to Choose
Three ways to lay out a Delta table so queries read less data — and the senior skill isn't picking the newest one, it's knowing when NOT to switch. Explained with a bookstore.
Read More
repartition vs coalesce in Spark: Which to Use and When
Both change the number of partitions — but one does a full shuffle and one doesn't, and that single difference decides your performance. Plus the coalesce(1) trap everyone falls into.
Read More
Fix Spark Data Skew in Joins (One Task Runs Forever)
Your job is "99% done" for an hour because one task got all the data. That's skew. Here's how to fix it with AQE, broadcast joins, skew hints, and salting.
Read More
Fix Over-Partitioned Delta Tables (Slow Queries) in Databricks
Partitioning by the wrong column shatters your table into millions of tiny files and grinds queries to a halt. Here's how to right-size it — and why liquid clustering is usually the better answer.
Read More