PySpark Data Serialization: Why Python UDFs Are Slow

You add a small Python function to your PySpark job. Your runtime triples. Nothing fails, nothing errors — it is just slow.

The cause is data serialization. This article covers what it is, why Python UDFs trigger it, how to detect it in the Spark UI, and how to fix it.

What is data serialization?

Data serialization is the process of converting data into a format that can be efficiently:

1. Saved to disk or storage — stored in a format that can be read back later.
2. Transmitted over the network — sent between nodes in a Spark cluster.
3. Transferred between processes — for example, moving data from a Scala JVM process to a Python process in PySpark.

The third one is what causes the problem in this article.

How PySpark actually runs

1. Spark is written in Scala and runs on the JVM (Java Virtual Machine).
2. When Spark reads data, it loads it into the JVM’s heap memory for processing.
3. PySpark is Spark’s Python API. It lets you write Spark code in Python.
4. The Spark driver translates your PySpark calls — filter, groupBy, withColumn and everything in pyspark.sql.functions — into operations the JVM already knows how to run.
5. So for normal PySpark code, all the work happens in the JVM. No data ever reaches Python.

This is why PySpark is fast, and why PySpark and Scala Spark perform almost identically on normal DataFrame operations.

What happens with a Python UDF

A Python UDF is your own Python function:

from pyspark.sql.functions import udf
from pyspark.sql.types import StringType

@udf(StringType())
def get_type(description):
    return description.split(",")[1]

df = df.withColumn("type", get_type("description"))

from pyspark.sql.functions import udf
from pyspark.sql.types import StringType

@udf(StringType())
def get_type(description):
    return description.split(",")[1]

df = df.withColumn("type", get_type("description"))

Now the rules change:

1. This is plain Python code. The Spark driver cannot translate it into a JVM operation, because there isn’t one.
2. Instead, Spark creates a Python runtime (a Python worker process) on the executor to run it.
3. Your data is in the JVM heap. Your function is in the Python process. The data has to move between them.

That movement is the serialization:

• Data is serialized when it moves from JVM heap memory into the Python process.
• After the Python function runs, the result is serialized back into a JVM-readable format.

In PySpark, this JVM-to-Python serialization only happens when you use a Python UDF. Normal DataFrame operations never trigger it.

The three costs

This back-and-forth introduces overhead in three places:

1. Serialization cost — Spark must convert JVM data into a Python-friendly format, using Pickle or Arrow.
2. Deserialization cost — after the Python code runs, the result must be converted back into a JVM-compatible format.
3. Transfer overhead — moving serialized data between the JVM and the Python runtime is expensive, especially for large datasets.

Problems with Python UDFs

1. Serialization with cloudpickle — Python UDFs serialize using cloudpickle, which is slow and adds significant overhead on large datasets. Every record is serialized on the way out and deserialized on the way back.
2. Record-by-record data transfer — data moves between the JVM and Python one record at a time. This fine-grained movement across processes creates high transfer and serialization overhead.
3. Record-by-record processing — the UDF also runs once per record, instead of using Spark’s vectorized operations. Lower throughput, longer runtime.

Detecting the problem in the Spark UI

If you use any Python UDF, Spark adds a node named BatchEvalPython to the query plan.

To find it: open the Spark UI, go to the SQL / DataFrame tab, and click your query.

Click the node to see the cost:

BatchEvalPython (3)
  data returned from Python workers total     4.9 GiB
  data sent to Python workers total          12.5 GiB
  rows output                             373,296,217

BatchEvalPython (3)
  data returned from Python workers total     4.9 GiB
  data sent to Python workers total          12.5 GiB
  rows output                             373,296,217

12.5 GiB serialized out of the JVM, every run.

Note that the name is misleading. BatchEvalPython contains the word “batch” but processes one row at a time. There are three nodes to know:

Ruling out other causes first

Before blaming a UDF, check the Stages tab. Serialization looks different from the other common problems.

This is what skew looks like:

The Max column dwarfs the Median — that is skew, and it needs a different fix.

The reason serialization hides from all three checks: the data moves between two processes on the same machine, not between machines. Shuffle metrics do not count it, memory is never the limit so nothing spills, and every task pays equally so nothing looks skewed.

Solution 1: use @pandas_udf

Replace @udf with @pandas_udf. That is the whole change.

import pandas as pd
from pyspark.sql.functions import pandas_udf

@pandas_udf("string")
def get_type(descriptions: pd.Series) -> pd.Series:
    return descriptions.str.split(",").str[1]

df = df.withColumn("type", get_type("description"))

import pandas as pd
from pyspark.sql.functions import pandas_udf

@pandas_udf("string")
def get_type(descriptions: pd.Series) -> pd.Series:
    return descriptions.str.split(",").str[1]

df = df.withColumn("type", get_type("description"))

How @pandas_udf works:

1. Arrow serialization — data is serialized in memory using Apache Arrow instead of row-by-row cloudpickle. Arrow is a shared memory format that both the JVM and Python understand, so transfer overhead drops sharply.
2. Vectorized operations — instead of one record at a time, the function receives data in batches as a pandas Series, and processes the whole batch at once.

After the change, the Spark UI shows ArrowEvalPython instead of BatchEvalPython:

Same rows in, same rows out, 12.5 GiB down to 4.5 MiB. Most of that 12.5 GiB was never your data — it was per-record serialization overhead repeated 373 million times.

Solution 2: avoid the UDF entirely

@pandas_udf reduces serialization. A built-in function removes it completely.

The example above just split a string and took the second element. Spark can already do that:

from pyspark.sql.functions import split, element_at

df = df.withColumn("type", element_at(split("description", ","), 2))

from pyspark.sql.functions import split, element_at

df = df.withColumn("type", element_at(split("description", ","), 2))

split and element_at are operations the JVM already knows, so no Python runtime is created and no serialization happens. The plan shows neither BatchEvalPython nor ArrowEvalPython.

There is a second benefit: Spark’s optimiser can see inside a built-in function and rearrange the query around it. It cannot see inside any UDF, including @pandas_udf.

Before writing a UDF, check pyspark.sql.functions. String handling, dates, JSON, regular expressions, conditional logic, arrays, maps, hashing and aggregation are all built in.

Solution 3: use Scala or Java UDFs

• Scala and Java UDFs execute natively on the JVM, so no serialization or deserialization is needed at all.
• They also benefit from Spark’s internal optimizations.
• The trade-off is maintaining JVM code alongside your Python codebase.

If your team is comfortable with Scala or Java, this is the fastest option of all for custom logic.

The three options compared

Tuning the batch size

When you do need @pandas_udf, you can control how many rows travel together:

# Rows per Arrow batch (default 10000)
spark.conf.get("spark.sql.execution.arrow.maxRecordsPerBatch")

spark.conf.set("spark.sql.execution.arrow.maxRecordsPerBatch", "20000")

# Rows per Arrow batch (default 10000)
spark.conf.get("spark.sql.execution.arrow.maxRecordsPerBatch")

spark.conf.set("spark.sql.execution.arrow.maxRecordsPerBatch", "20000")

• Larger batches mean fewer transfers and less overhead.
• Larger batches also need more memory in the Python worker.
• That Python memory sits outside the JVM heap, so executor memory settings do not control it. If Python workers crash on wide tables, lower this value rather than resizing the cluster.

Quick reference

Summary

1. Spark runs on the JVM and keeps your data in JVM heap memory.
2. Normal PySpark operations are translated into JVM operations, so no data reaches Python.
3. A Python UDF cannot be translated, so Spark creates a Python runtime on the executor and serializes data to it — one record at a time.
4. Spark marks this in the query plan as BatchEvalPython.
5. @pandas_udf switches to Arrow serialization and batch processing, shown as ArrowEvalPython. In the job above that reduced data sent from 12.5 GiB to 4.5 MiB.
6. A built-in Spark function removes the serialization entirely, and is always the first thing to check for.

For more on how drivers and executors fit together, see Spark Architecture Explained Simply. For the built-in operations worth reaching for first, see the PySpark DataFrame guide.

Frequently Asked Questions

What is data serialization in Spark?

Converting data into a format that can be saved to disk, sent over a network, or transferred between processes. In PySpark the costly case is the third one: moving data from the Scala JVM process into a Python process so a Python UDF can run on it, then converting the result back.

When does PySpark need to serialize data?

The JVM-to-Python serialization only happens when you use a Python UDF. Normal DataFrame operations like filter, groupBy and withColumn are translated into JVM operations, so the data never leaves the JVM heap and nothing is serialized.

What is BatchEvalPython in the Spark UI?

The node Spark adds to the query plan when a Python UDF runs. Despite the name, it processes one record at a time — each is serialized with cloudpickle, sent to a Python worker, processed, and sent back. Click it and read “data sent to Python workers” to see the cost.

What is the difference between udf and pandas_udf?

@udf serializes with cloudpickle one record at a time and calls your function once per record. @pandas_udf serializes with Apache Arrow in batches and hands your function a pandas Series to process all at once. In the example above, the same job sent 12.5 GiB with @udf and 4.5 MiB with @pandas_udf.

Why is my Spark stage slow with no shuffle and no spill?

Because the cost is inside the task, not in moving data between machines. Serialization moves data between two processes on the same machine, which shuffle metrics do not count; memory is never the limit so nothing spills; and every task pays equally so nothing looks skewed. Check the SQL / DataFrame tab for BatchEvalPython.

Are Scala UDFs faster than Python UDFs?

Yes. Scala and Java UDFs execute natively on the JVM, so no serialization or deserialization is required at all, and they benefit from Spark’s internal optimizations. The trade-off is maintaining JVM code alongside your Python codebase.

Do I need to install Apache Arrow to use pandas_udf?

PyArrow is required, and it comes preinstalled in Databricks, Microsoft Fabric and most managed Spark environments. On a self-managed cluster you may need to install pyarrow on every node, not just the driver.

My Python workers keep crashing. Should I add more executor memory?

Usually not. Python worker memory sits outside the JVM heap, so executor memory settings do not govern it. With @pandas_udf each worker holds a full batch at once, which adds up on wide tables. Lower spark.sql.execution.arrow.maxRecordsPerBatch from its default of 10000 first.