Fix Corrupted Parquet Files in Spark (“Could Not Read Footer”)
One bad file shouldn't take down the whole read. Here's why Parquet files corrupt, how to skip past them to get your data now, and how to find and fix the real culprit.
Read MoreTag: Data Quality
Why Empty Strings Become Null in a Partitioned Column (Spark)
No error, just wrong data: partition by a column with empty strings and Spark reads them back as null. Here's exactly why — and how to keep the values distinct.
Read More