The Small Files Problem in Spark: How to Detect and Fix It
41,600 files holding 14.8 GB took 34 minutes just to scan. Here is what causes the small files problem, the exact numbers that reveal it in the Spark UI, and four ways to fix it.
Read MoreTag: Delta Lake
Full Recompute vs Incremental Processing: Which to Use When
Everyone assumes incremental is always the better choice. It isn't — for small tables it's over-engineering. Here's the real trade-off, a simple decision tree, and how to detect changes.
Read More
MERGE vs INSERT OVERWRITE vs APPEND: How to Write a Delta Table
Reaching for MERGE by default is one of the most expensive habits in data engineering. The right write pattern depends on your source — here's the decision, one question at a time.
Read More
Data Ingestion Patterns: How to Choose the Right Way to Load Data
Getting data in seems simple until the bill arrives or the SLA is missed. Two questions — what's the source, and how fresh must it be — decide the right pattern every time.
Read More
Liquid Clustering vs Z-ORDER vs Partitioning: Which to Choose
Three ways to lay out a Delta table so queries read less data — and the senior skill isn't picking the newest one, it's knowing when NOT to switch. Explained with a bookstore.
Read More
How Delta Lake Does ACID Without a Database
No database server, no lock manager — just files and a log. Here's how two Spark jobs can write the same table at the exact same moment and never corrupt it.
Read More
Delta Table Internals Explained: Files, Transaction Log, ACID & Time Travel
Follow one Delta table from CREATE to time travel and watch exactly what happens on disk — the Parquet files, the transaction log, and how that log quietly powers ACID and versioning.
Read More
Medallion Architecture Explained Simply: Bronze, Silver & Gold
Raw data goes in one end, business-ready data comes out the other — refined in three clear stages. Here's the Bronze, Silver, and Gold pattern (plus the semantic layer), traced through a real production pipeline.
Read More