Home › Learn › Shift left › Tests, coverage, complexity
Shift left · page 5 of 13
Unit tests, coverage and complexity for PySpark
Unit tests for PySpark as a table of cases, coverage.py and its blind spots, coverage on changed lines, mutation testing, and complexity scores.
Unit tests
A table of cases, a tiny DataFrame, seconds to run
A pure function is tested by building a few rows on a local Spark session and comparing the result with Spark's built-in assertDataFrameEqual (Spark 3.5 and later). Write the cases as one table: each row is a name, the input rows and the expected rows. Adding a case is adding a line.
import pytest
from pyspark.testing import assertDataFrameEqual
from silver.orders import dedupe_orders
SCHEMA = "order_id INT, amount DOUBLE"
CASES = [
# name, input rows, expected rows
("removes duplicate orders", [(1, 10.0), (1, 10.0)], [(1, 10.0)]),
("keeps distinct orders", [(1, 10.0), (2, 5.0)], [(1, 10.0), (2, 5.0)]),
("empty input stays empty", [], []),
]
@pytest.mark.parametrize("name, rows, expected", CASES, ids=[c[0] for c in CASES])
def test_dedupe_orders(spark, name, rows, expected):
got = dedupe_orders(spark.createDataFrame(rows, SCHEMA))
assertDataFrameEqual(got, spark.createDataFrame(expected, SCHEMA))Run on Spark 3.5.1 on 2 Nov 2026: all cases pass in about 7 seconds. The spark fixture is one local session shared by all tests.
Two setup traps we hit running this. In Spark 3.5, importing pyspark.testing needs pandas and pyarrow installed (and numpy below 2 with older pandas). And Spark's worker processes must use the same Python as the test: set PYSPARK_PYTHON to the test's Python, or the tests fail with a Python version mismatch.
Coverage
coverage.py works for PySpark too
Python's standard coverage tool is coverage.py, usually run through pytest-cov. It measures PySpark the same way as any Python, because PySpark code is Python running on the driver. It reports which lines and which branches of each if ran during the tests.
diff-cover then looks only at the lines a pull request changed, so the gate can be "80% of the changed lines are covered" without failing on years of old code.
| Blind spot | Why | What to do |
|---|---|---|
| Code inside Python UDFs | It runs in separate worker processes | Test the UDF's Python function directly, or configure coverage for subprocesses |
| SQL text | SQL has no lines that coverage can count | Test SQL by checking its output on tiny tables |
| Notebooks | Hard to measure and import | Put the logic in .py modules; keep notebooks as thin callers |
The purity rule fixes all three: the logic sits in pure functions in modules, where coverage works.
Do the tests bite?
Mutation testing
Coverage says a line ran, not that a test would notice if it were wrong. mutmut changes the code on purpose (> becomes >=, a column name is swapped) and checks that some test fails. The share of changes caught is the mutation score. It is slow, so it runs nightly, on the pure functions only. Hypothesis adds property-based tests that try many generated inputs against a rule such as "never more rows out than in".
Complexity score
How tangled is each function?
| Score | What it counts | Tool | Verdict |
|---|---|---|---|
| Cyclomatic complexity | The number of paths through a function (each if, loop, and/or adds one) | Ruff rule C901 (McCabe) | Pursue |
| Cognitive complexity | How hard it is to read: nesting costs extra, so deep ifs score high | complexipy (fast, MIT licence) | Pursue |
| Maintainability index, trends | A combined score; history over git | radon, wily | Drop: no releases since 2023–2024 |
| Many languages | Complexity for Scala, Java, SQL and others | lizard | Later, when other languages arrive |
The useful view joins the three reports: complex and untested functions are where bugs hide, and pure but untested functions are the cheapest tests to add.
PURITY 112 functions: 61 pure, 51 impure (29 of them could be pure) TESTS 17 pure functions have no test · changed lines covered: 64% (gate 80%) FAIL COMPLEXITY gold/kpis.py:build_daily_kpis cognitive 31, cyclomatic 18, untested <- fix first
Illustrative report, not measured on a real repo.
FAQ
Questions people ask
Does Python have a code coverage tool?
Yes. coverage.py is the standard; pytest-cov runs it with pytest. It reports line and branch coverage, and diff-cover limits the check to the lines a change touched.
How do I measure test coverage for PySpark?
The same way as Python: coverage.py via pytest-cov, with a local Spark session in the tests. It doesn't see code inside Python UDFs running on workers, SQL text, or notebooks, so keep logic in pure functions in modules.
Is there a tool that gives a code complexity score for Python?
Yes. Ruff's C901 rule gives cyclomatic complexity, and complexipy gives cognitive complexity, which punishes nesting and matches readability better.
Next
Where to go next
Next page: Testing SCD2, idempotency, duplicates and dropped rows without a cluster
↑ All of shift left · The repo on GitHub · Data project review