Home › Learn › Shift left › Tests, coverage, complexity

Shift left · page 5 of 13

Unit tests, coverage and complexity for PySpark

Unit tests for PySpark as a table of cases, coverage.py and its blind spots, coverage on changed lines, mutation testing, and complexity scores.

↑ All of shift leftShift left, explainedHow software solved itSyntax treesPure functionsTests, coverage, complexityTesting data logic offlineClean codeSecurity and costWhat is missingThe bundleHow it is builtAll the checksOpen source and ours

Unit tests

A table of cases, a tiny DataFrame, seconds to run

A pure function is tested by building a few rows on a local Spark session and comparing the result with Spark's built-in assertDataFrameEqual (Spark 3.5 and later). Write the cases as one table: each row is a name, the input rows and the expected rows. Adding a case is adding a line.

import pytest
from pyspark.testing import assertDataFrameEqual
from silver.orders import dedupe_orders

SCHEMA = "order_id INT, amount DOUBLE"
CASES = [
    # name,                     input rows,                 expected rows
    ("removes duplicate orders", [(1, 10.0), (1, 10.0)],     [(1, 10.0)]),
    ("keeps distinct orders",    [(1, 10.0), (2, 5.0)],      [(1, 10.0), (2, 5.0)]),
    ("empty input stays empty",  [],                         []),
]

@pytest.mark.parametrize("name, rows, expected", CASES, ids=[c[0] for c in CASES])
def test_dedupe_orders(spark, name, rows, expected):
    got = dedupe_orders(spark.createDataFrame(rows, SCHEMA))
    assertDataFrameEqual(got, spark.createDataFrame(expected, SCHEMA))

Run on Spark 3.5.1 on 2 Nov 2026: all cases pass in about 7 seconds. The spark fixture is one local session shared by all tests.

Two setup traps we hit running this. In Spark 3.5, importing pyspark.testing needs pandas and pyarrow installed (and numpy below 2 with older pandas). And Spark's worker processes must use the same Python as the test: set PYSPARK_PYTHON to the test's Python, or the tests fail with a Python version mismatch.

Coverage

coverage.py works for PySpark too

Python's standard coverage tool is coverage.py, usually run through pytest-cov. It measures PySpark the same way as any Python, because PySpark code is Python running on the driver. It reports which lines and which branches of each if ran during the tests.

diff-cover then looks only at the lines a pull request changed, so the gate can be "80% of the changed lines are covered" without failing on years of old code.

Blind spotWhyWhat to do
Code inside Python UDFsIt runs in separate worker processesTest the UDF's Python function directly, or configure coverage for subprocesses
SQL textSQL has no lines that coverage can countTest SQL by checking its output on tiny tables
NotebooksHard to measure and importPut the logic in .py modules; keep notebooks as thin callers

The purity rule fixes all three: the logic sits in pure functions in modules, where coverage works.

Do the tests bite?

Mutation testing

Coverage says a line ran, not that a test would notice if it were wrong. mutmut changes the code on purpose (> becomes >=, a column name is swapped) and checks that some test fails. The share of changes caught is the mutation score. It is slow, so it runs nightly, on the pure functions only. Hypothesis adds property-based tests that try many generated inputs against a rule such as "never more rows out than in".

Complexity score

How tangled is each function?

ScoreWhat it countsToolVerdict
Cyclomatic complexityThe number of paths through a function (each if, loop, and/or adds one)Ruff rule C901 (McCabe)Pursue
Cognitive complexityHow hard it is to read: nesting costs extra, so deep ifs score highcomplexipy (fast, MIT licence)Pursue
Maintainability index, trendsA combined score; history over gitradon, wilyDrop: no releases since 2023–2024
Many languagesComplexity for Scala, Java, SQL and otherslizardLater, when other languages arrive

The useful view joins the three reports: complex and untested functions are where bugs hide, and pure but untested functions are the cheapest tests to add.

PURITY      112 functions: 61 pure, 51 impure (29 of them could be pure)
TESTS       17 pure functions have no test · changed lines covered: 64% (gate 80%)  FAIL
COMPLEXITY  gold/kpis.py:build_daily_kpis  cognitive 31, cyclomatic 18, untested  <- fix first

Illustrative report, not measured on a real repo.

FAQ

Questions people ask

Does Python have a code coverage tool?

Yes. coverage.py is the standard; pytest-cov runs it with pytest. It reports line and branch coverage, and diff-cover limits the check to the lines a change touched.

How do I measure test coverage for PySpark?

The same way as Python: coverage.py via pytest-cov, with a local Spark session in the tests. It doesn't see code inside Python UDFs running on workers, SQL text, or notebooks, so keep logic in pure functions in modules.

Is there a tool that gives a code complexity score for Python?

Yes. Ruff's C901 rule gives cyclomatic complexity, and complexipy gives cognitive complexity, which punishes nesting and matches readability better.

Next

Where to go next

Next page: Testing SCD2, idempotency, duplicates and dropped rows without a cluster

↑ All of shift left · The repo on GitHub · Data project review