Home › Learn › Shift left › Pure functions
Shift left · page 4 of 13
Pure functions in PySpark: possible, and how we detect them
Pure functions in PySpark: what they are, how to refactor impure code into a pure core and thin shell, and how a tool classifies functions as pure, impure or unknown.
In plain words
What a pure function is
A pure function gives the same output for the same input and touches nothing outside itself: no reading a table, no writing, no clock, no random numbers, no secrets. Pure functions are the easiest code to test, because a test only has to pass data in and check what comes out.
Spark makes this easier than plain Python. A DataFrame never changes in place; every step returns a new one, and nothing runs until an action such as a write. So a function that takes a DataFrame and returns a DataFrame using only column expressions is pure. Spark even has df.transform(f) to chain such functions.
Before and after
Turning an impure function into a pure one
# Before: impure. It reads a table, uses the clock and writes, so a test needs real tables.
def load_silver_orders():
df = spark.table("bronze.orders")
df = df.dropDuplicates(["order_id"]).withColumn("load_ts", F.lit(datetime.now()))
df.write.mode("append").saveAsTable("silver.orders")
# After: the logic is pure; the read, the clock and the write sit in a thin shell.
def dedupe_orders(orders: DataFrame) -> DataFrame: # pure
return orders.dropDuplicates(["order_id"])
def add_load_ts(df: DataFrame, load_ts: datetime) -> DataFrame: # pure: the time is passed in
return df.withColumn("load_ts", F.lit(load_ts))
def run(spark): # impure shell
orders = spark.table("bronze.orders")
out = orders.transform(dedupe_orders).transform(add_load_ts, datetime.now())
out.write.mode("append").saveAsTable("silver.orders")
What stays impure, on purpose: reads, writes, MERGE, dbutils and job entry points. Keep them thin. For a MERGE, the "what to merge" logic (removing duplicates, building history rows) is pure and only the final MERGE call is not. Values like current_timestamp(), rand() and uuid() make a function give different answers each run, so pass them in. Keep actions such as .count() and .collect() out of pure functions too: they make Spark run, and running costs money.
Can a tool tell?
How we detect a pure function
The tool reads each function's syntax tree and looks for anything that touches the world outside it. Python is too dynamic to prove purity in every case, so it gives three answers instead of two.
| Verdict | Meaning | Examples |
|---|---|---|
| impure | A side effect was found. The report shows the reason and the line. | spark.table, .write, saveAsTable, DeltaTable.merge, dbutils, open, HTTP calls, os.environ, datetime.now, random, current_timestamp(), rand(), uuid(), a global spark, changing an argument or a global, actions such as collect(), count(), show() |
| pure | Every call is on a known-pure list and nothing outside is touched. | DataFrame methods that return a new DataFrame (select, withColumn, filter, join, groupBy().agg, dropDuplicates, union), most column functions, math, re, and functions in the repo already found pure |
| unknown | It calls something the tool can't see into. Counted as not pure until someone marks it. | a third-party library call, a function passed in as an argument, getattr tricks |
- It follows calls. If
dedupe_orderscalls a helper that reads a table, both are impure, and the report shows the chain. - Marking. A developer can mark a function
@pure, using the marker from the open-source deal library. The static check still runs, so a wrong mark is reported. - Proof at test time. A test fixture makes Spark's read, write and catalog calls fail during unit tests of pure functions. If a "pure" function touches a table, its test fails. Each test also runs the function twice and checks the outputs match.
- Measured, not claimed. A sample repo with planted pure, impure and tricky functions, and a hidden answer key, tells us how often the verdict is right.
The report then lists: how many functions are pure, which impure ones could be pure (with a hint such as "take the DataFrame as a parameter; move the read to the caller"), and which pure ones have no test. See tests and coverage.
FAQ
Questions people ask
Can PySpark code be written as pure functions?
Yes, for most of the logic. DataFrames are immutable, so a function that takes a DataFrame and returns one using column expressions is pure. Reads, writes, MERGE calls and dbutils stay in a thin outer layer.
How can a tool tell whether a Python function is pure?
It reads the syntax tree and looks for side effects: table reads and writes, dbutils, file and network calls, the clock, random values, globals and Spark actions. Calls it can't see into are reported as unknown, not as pure.
Is current_timestamp() pure in Spark?
No. It gives a different value on each run, so a function using it can't be tested with a fixed expected result. Pass the timestamp in as a parameter.
Next
Where to go next
Next page: Unit tests, coverage and complexity for PySpark
↑ All of shift left · The repo on GitHub · Data project review