Home › Learn › Shift left › Pure functions

Shift left · page 4 of 13

Pure functions in PySpark: possible, and how we detect them

Pure functions in PySpark: what they are, how to refactor impure code into a pure core and thin shell, and how a tool classifies functions as pure, impure or unknown.

↑ All of shift leftShift left, explainedHow software solved itSyntax treesPure functionsTests, coverage, complexityTesting data logic offlineClean codeSecurity and costWhat is missingThe bundleHow it is builtAll the checksOpen source and ours

In plain words

What a pure function is

A pure function gives the same output for the same input and touches nothing outside itself: no reading a table, no writing, no clock, no random numbers, no secrets. Pure functions are the easiest code to test, because a test only has to pass data in and check what comes out.

Spark makes this easier than plain Python. A DataFrame never changes in place; every step returns a new one, and nothing runs until an action such as a write. So a function that takes a DataFrame and returns a DataFrame using only column expressions is pure. Spark even has df.transform(f) to chain such functions.

impure shell: thin, tested by integration testsReadspark.tablepure coreDataFrame in, DataFrame outunit tests in secondsWritesaveAsTableclock, random, dbutils and secrets also stay in the shell and are passed in
Most of the logic lives in the pure core. The shell around it is thin.

Before and after

Turning an impure function into a pure one

# Before: impure. It reads a table, uses the clock and writes, so a test needs real tables.
def load_silver_orders():
    df = spark.table("bronze.orders")
    df = df.dropDuplicates(["order_id"]).withColumn("load_ts", F.lit(datetime.now()))
    df.write.mode("append").saveAsTable("silver.orders")


# After: the logic is pure; the read, the clock and the write sit in a thin shell.
def dedupe_orders(orders: DataFrame) -> DataFrame:                      # pure
    return orders.dropDuplicates(["order_id"])

def add_load_ts(df: DataFrame, load_ts: datetime) -> DataFrame:         # pure: the time is passed in
    return df.withColumn("load_ts", F.lit(load_ts))

def run(spark):                                                         # impure shell
    orders = spark.table("bronze.orders")
    out = orders.transform(dedupe_orders).transform(add_load_ts, datetime.now())
    out.write.mode("append").saveAsTable("silver.orders")

What stays impure, on purpose: reads, writes, MERGE, dbutils and job entry points. Keep them thin. For a MERGE, the "what to merge" logic (removing duplicates, building history rows) is pure and only the final MERGE call is not. Values like current_timestamp(), rand() and uuid() make a function give different answers each run, so pass them in. Keep actions such as .count() and .collect() out of pure functions too: they make Spark run, and running costs money.

Can a tool tell?

How we detect a pure function

The tool reads each function's syntax tree and looks for anything that touches the world outside it. Python is too dynamic to prove purity in every case, so it gives three answers instead of two.

pureonly known-pure callsDataFrame in, outunit test itimpurefound a side effectreason + line shownkeep it thinunknowncalls something unseencounts as not pureuntil marked @pure
Unknown is honest: the tool says when it can't see far enough.
VerdictMeaningExamples
impureA side effect was found. The report shows the reason and the line.spark.table, .write, saveAsTable, DeltaTable.merge, dbutils, open, HTTP calls, os.environ, datetime.now, random, current_timestamp(), rand(), uuid(), a global spark, changing an argument or a global, actions such as collect(), count(), show()
pureEvery call is on a known-pure list and nothing outside is touched.DataFrame methods that return a new DataFrame (select, withColumn, filter, join, groupBy().agg, dropDuplicates, union), most column functions, math, re, and functions in the repo already found pure
unknownIt calls something the tool can't see into. Counted as not pure until someone marks it.a third-party library call, a function passed in as an argument, getattr tricks

The report then lists: how many functions are pure, which impure ones could be pure (with a hint such as "take the DataFrame as a parameter; move the read to the caller"), and which pure ones have no test. See tests and coverage.

FAQ

Questions people ask

Can PySpark code be written as pure functions?

Yes, for most of the logic. DataFrames are immutable, so a function that takes a DataFrame and returns one using column expressions is pure. Reads, writes, MERGE calls and dbutils stay in a thin outer layer.

How can a tool tell whether a Python function is pure?

It reads the syntax tree and looks for side effects: table reads and writes, dbutils, file and network calls, the clock, random values, globals and Spark actions. Calls it can't see into are reported as unknown, not as pure.

Is current_timestamp() pure in Spark?

No. It gives a different value on each run, so a function using it can't be tested with a fixed expected result. Pass the timestamp in as a parameter.

Next

Where to go next

Next page: Unit tests, coverage and complexity for PySpark

↑ All of shift left · The repo on GitHub · Data project review