Home › Learn › How reads and writes work › 6 · Python and JVM
Delta Lake and Spark · from zero to advanced
PySpark: what runs in Python, what runs in the JVM
How PySpark really works: the Python process, the Py4J bridge to the JVM, what the DataFrame API builds, what a Python UDF costs (measured), what collect converts, and why built-in functions are fast. Real process ids and timings.
Two processes
Python is the remote control, the JVM is the engine
PySpark is not Spark rewritten in Python. Spark runs on the JVM. Your Python program starts that JVM and sends it instructions through a bridge called Py4J. When you write df.filter(...) you are not filtering anything: you are asking the JVM to add a step to its plan.
Two separate processes, and extra Python workers only when Python code must run on the data.
DataFrame object wrapped a JVM org.apache.spark.sql.Dataset.What stays in the JVM
Built-in functions never leave the JVM
filter, select, join, groupBy and the functions in pyspark.sql.functions are all instructions that the JVM executes itself. No data is sent to Python. Their speed is the same as Scala or SQL, because the work is the same.
| You write | Runs in | Python sees data? |
|---|---|---|
df.filter(F.col('amount') > 800) | JVM | No |
F.col('amount') * 1.1 | JVM | No |
df.groupBy(...).agg(F.sum(...)) | JVM | No |
udf(lambda x: x * 1.1) | Python workers | Yes, every value |
df.rdd.map(lambda r: ...) | Python workers | Yes, every row |
df.collect() / df.toPandas() | JVM, then driver Python | Yes, all rows |
The cost of a Python UDF
Same arithmetic, written two ways
Every row makes a round trip to a Python process.
BatchEvalPython node. An equivalent rdd.map(lambda …) took 0.36 s. Treat the ratio as indicative: on other data and hardware it will differ.- Why slower: each value is copied out of the JVM, run through Python, and copied back. The optimizer also cannot push a filter through the lambda or fuse it with neighbouring steps.
- Arrow and pandas UDFs move data in column batches instead of row by row and are the usual faster option, but they need PyArrow, which was not installed in the test environment, so they were not measured.
- The fix is usually built-in functions. Check
pyspark.sql.functionsbefore writing a UDF.
Conversion at the edge
What collect() converts, and why it is slow
The JVM holds rows in its own format. collect() must serialise every row, send it to the driver process, and build a Python object for each one. That is conversion work on top of the network transfer.
| Action | Result size | Time | Why |
|---|---|---|---|
groupBy.agg().collect() | 21 KB | 0.14 s | A few summary rows |
collect(), 2 columns | 2.2 MB | 0.78 s | 200,000 small rows to Python |
collect(), 40 columns | 33.1 MB | 1.97 s | Same rows, 15× the bytes |
The rule: aggregate or filter in Spark, and collect only the small result.
What to do
Python and JVM rules
| Do | Verdict |
|---|---|
| Use built-in functions and SQL expressions | Pursue |
| Use Arrow or pandas UDFs when you must run Python | Pursue |
| Collect only small, aggregated results | Pursue |
| Write a row-by-row Python UDF for something a built-in can do | Drop |
| Loop over a collected DataFrame in Python | Drop |
FAQ
Questions people ask
Does PySpark process data in Python?
No. Your Python program sends instructions to the Spark JVM over Py4J, and the JVM runs the plan. Python only touches data if you use a UDF, an RDD lambda, or collect results.
How much slower is a Python UDF?
About 2 times slower than the equivalent built-in expression in the local test, 0.25 seconds against 0.13 seconds for 200,000 rows, and the plan shows a BatchEvalPython step. The gap grows with more data.
What is the difference between a Python UDF and a pandas UDF?
A Python UDF handles one row at a time. A pandas UDF (Arrow-based) handles column batches, which reduces conversion cost. Pandas UDFs need PyArrow, which was not available in the local test, so only the plain UDF was measured.