Home › Learn › How reads and writes work › 6 · Python and JVM

Delta Lake and Spark · from zero to advanced

PySpark: what runs in Python, what runs in the JVM

How PySpark really works: the Python process, the Py4J bridge to the JVM, what the DataFrame API builds, what a Python UDF costs (measured), what collect converts, and why built-in functions are fast. Real process ids and timings.

Two processes

Python is the remote control, the JVM is the engine

PySpark is not Spark rewritten in Python. Spark runs on the JVM. Your Python program starts that JVM and sends it instructions through a bridge called Py4J. When you write df.filter(...) you are not filtering anything: you are asking the JVM to add a step to its plan.

Your Python processthe driver programprocess id 80699df = spark.read…The JVM (Spark driver)process id 80703builds the plan, runs thescheduler, holds the log statePy4JExecutor JVM tasks…do the real workPython workersonly for UDFsids 80870, 80871Your Python code never touches the data. It sends instructions to the JVM over a socket (Py4J).The DataFrame object in Python is a thin handle to a JVM object of type org.apache.spark.sql.Dataset.

Two separate processes, and extra Python workers only when Python code must run on the data.

Measured. Real process ids from the run: the Python driver program was process 80699, the Spark JVM was 80703 (Java 17.0.18), and the Python workers that ran a UDF were 80870 and 80871. The Python DataFrame object wrapped a JVM org.apache.spark.sql.Dataset.

What stays in the JVM

Built-in functions never leave the JVM

filter, select, join, groupBy and the functions in pyspark.sql.functions are all instructions that the JVM executes itself. No data is sent to Python. Their speed is the same as Scala or SQL, because the work is the same.

You writeRuns inPython sees data?
df.filter(F.col('amount') > 800)JVMNo
F.col('amount') * 1.1JVMNo
df.groupBy(...).agg(F.sum(...))JVMNo
udf(lambda x: x * 1.1)Python workersYes, every value
df.rdd.map(lambda r: ...)Python workersYes, every row
df.collect() / df.toPandas()JVM, then driver PythonYes, all rows

The cost of a Python UDF

Same arithmetic, written two ways

JVM executorrows in columnaror row formSerialise rows(pickle, in batches)rows outPython workerruns your lambdarow by rowJVM continueswith the resultDeserialiseresults backThe costevery value iscopied out of theJVM and back,and the optimizercannot see insideyour lambda

Every row makes a round trip to a Python process.

Measured. Multiplying a column by 1.1 on 200,000 rows: the built-in expression took 0.13 s and its plan had no Python step; the Python UDF took 0.25 s (2.0× slower) and its plan contained a BatchEvalPython node. An equivalent rdd.map(lambda …) took 0.36 s. Treat the ratio as indicative: on other data and hardware it will differ.
  • Why slower: each value is copied out of the JVM, run through Python, and copied back. The optimizer also cannot push a filter through the lambda or fuse it with neighbouring steps.
  • Arrow and pandas UDFs move data in column batches instead of row by row and are the usual faster option, but they need PyArrow, which was not installed in the test environment, so they were not measured.
  • The fix is usually built-in functions. Check pyspark.sql.functions before writing a UDF.

Conversion at the edge

What collect() converts, and why it is slow

The JVM holds rows in its own format. collect() must serialise every row, send it to the driver process, and build a Python object for each one. That is conversion work on top of the network transfer.

ActionResult sizeTimeWhy
groupBy.agg().collect()21 KB0.14 sA few summary rows
collect(), 2 columns2.2 MB0.78 s200,000 small rows to Python
collect(), 40 columns33.1 MB1.97 sSame rows, 15× the bytes

The rule: aggregate or filter in Spark, and collect only the small result.

What to do

Python and JVM rules

DoVerdict
Use built-in functions and SQL expressionsPursue
Use Arrow or pandas UDFs when you must run PythonPursue
Collect only small, aggregated resultsPursue
Write a row-by-row Python UDF for something a built-in can doDrop
Loop over a collected DataFrame in PythonDrop

FAQ

Questions people ask

Does PySpark process data in Python?

No. Your Python program sends instructions to the Spark JVM over Py4J, and the JVM runs the plan. Python only touches data if you use a UDF, an RDD lambda, or collect results.

How much slower is a Python UDF?

About 2 times slower than the equivalent built-in expression in the local test, 0.25 seconds against 0.13 seconds for 200,000 rows, and the plan shows a BatchEvalPython step. The gap grows with more data.

What is the difference between a Python UDF and a pandas UDF?

A Python UDF handles one row at a time. A pandas UDF (Arrow-based) handles column batches, which reduces conversion cost. Pandas UDFs need PyArrow, which was not available in the local test, so only the plain UDF was measured.

← 5 · Inside Spark7 · Photon →