Databricks Troubleshooting

Why Empty Strings Become Null in a Partitioned Column (Spark)

No error, just wrong data: partition by a column with empty strings and Spark reads them back as null. Here's exactly why — and how to keep the values distinct.

Databricks Troubleshooting

This one is dangerous precisely because nothing breaks. You write a DataFrame partitioned by a column, some of whose values are empty strings (""). The job succeeds. Everyone moves on. Then weeks later a report is quietly wrong, because when you read the data back, those empty strings have silently turned into null.

No exception, no warning — just changed data. Here’s exactly where it happens.

⚡ Quick fix

Replace empty strings (and nulls, if you need to tell them apart later) with a real sentinel value before you partition:

from pyspark.sql.functions import when, col, lit

clean = df.withColumn(
    "region",
    when((col("region").isNull()) | (col("region") == ""), lit("UNKNOWN"))
    .otherwise(col("region"))
)

clean.write.partitionBy("region").parquet("/mnt/data/sales")

Now every partition value is a real, distinct string, so nothing collapses into null on read. If you must preserve empty-string and null as separate categories, map them to two different sentinels (e.g. "EMPTY" and "UNKNOWN").

What actually happens

Partition values aren’t stored inside the files — they’re encoded in the folder names. When you partition by region, Spark writes a directory per value: region=East, region=West, and so on. The value is read back by parsing the folder name.

That’s where the trouble starts. A null value has no folder name to write, so Spark puts it in a special directory called region=__HIVE_DEFAULT_PARTITION__. And an empty string — which also can’t form a meaningful folder name — collapses into that same default partition. On read, both come back as null. The empty string is simply gone.

You write Folder on disk You read back “East” “” (empty string) null region=East region=__HIVE_DEFAULT_PARTITION__ empty AND null both land here “East” null null The surprise: the empty string collapses into the default partition and reads back as null.

You can see it directly in the folder listing after a write — there’s no region= folder for the empty string; it went into the default partition alongside the nulls:

/mnt/data/sales/region=East/part-00000.parquet
/mnt/data/sales/region=West/part-00000.parquet
/mnt/data/sales/region=__HIVE_DEFAULT_PARTITION__/part-00000.parquet   <- empty strings AND nulls

Why it happens

  • Partition values live in directory names, not in the data. A folder name can’t cleanly represent “empty string,” so it isn’t preserved.
  • Null and empty string share the default partition. Both route to __HIVE_DEFAULT_PARTITION__, erasing the distinction between them.
  • Reading parses the folder name back into a value, and the default partition always maps to null — so an empty string can never survive the round trip.

This isn’t a bug you can flag on with a config — it’s a fundamental consequence of how folder-based partitioning encodes values. The fix is to not put ambiguous values into a partition column in the first place.

How to verify it’s fixed

# After writing the cleaned data, read it back and check
result = spark.read.parquet("/mnt/data/sales")
result.select("region").distinct().show()

# You should see UNKNOWN (or your sentinel) as a real value,
# not a mysterious null where empty strings used to be.

How to prevent it

  • Don’t partition by free-text or nullable columns. Partition keys should be clean, low-cardinality, and never empty — think date, country, event_type.
  • Clean partition columns before writing — replace empty/null with an explicit sentinel so the value is unambiguous.
  • Keep the original column too if you need the exact raw value. Partition by a cleaned copy, but store the true value as a normal (non-partition) column inside the data, where empty strings are preserved.

When it’s actually something else

  • The data was already null upstream. Confirm the source actually had empty strings, not nulls, before blaming the write.
  • Too many partitions: if the real issue is a partition column with thousands of distinct values slowing everything down, that’s over-partitioning — a separate performance problem.
  • Non-partition columns are fine. Empty strings are only lost in partition columns. A regular column preserves "" and null distinctly.

For the official reference, see the Databricks knowledge base; the guidance above comes from debugging real partitioned pipelines.

Frequently Asked Questions

Why do empty strings become null in a partitioned column?

Partition values are stored as directory names, not inside the data. An empty string cannot form a meaningful folder name, so Spark routes it to the default partition (__HIVE_DEFAULT_PARTITION__) along with nulls. On read, that directory maps back to null, so the empty string is lost.

How do I keep empty strings in a partition column?

Replace empty strings with an explicit sentinel value before writing, such as “EMPTY” or “UNKNOWN”. Because the partition value is then a real, distinct string, it survives the round trip. If empty and null must stay separate, map them to two different sentinels.

Does this affect non-partition columns too?

No. Empty strings are only lost in partition columns, because those values are encoded in folder names. A regular, non-partition column stores empty strings and nulls distinctly inside the data files, so both are preserved on read.

What is __HIVE_DEFAULT_PARTITION__?

It is the special directory name Spark and Hive use to store rows whose partition value is null (or otherwise cannot be represented as a folder name, like an empty string). When the data is read, that directory is interpreted as a null partition value.

What columns are safe to partition by?

Partition by clean, low-cardinality columns that are never empty or null, such as date, country, or event type. Avoid free-text or nullable columns, which cause both correctness issues (like this one) and performance problems from too many partitions.

Related Posts: