Home › Learn › Shift left

Data engineering · free guide · 13 short pages

Shift left: fail fast on data code

How to catch problems in PySpark, Python and Databricks SQL before the code runs: cost, security, vulnerable libraries, untested logic, tangled functions, and what a repo is missing. Built from open-source tools the software world already trusts, with no heavy servers.

The picture

Find it on the left, where it is cheap

shift-left checks work here: no cluster, no dataEditorsecondsCommitunder 10 sPull requestunder 5 minPre-deployunder 1 minTest runhoursProductiona failed jobThe billmoney spentthe cost of finding a problem goes up this way
Shift-left checks live in the dashed box: no cluster, no data, seconds to minutes.

The short answer: make most of the logic pure functions (DataFrame in, DataFrame out) so it can be unit tested in seconds. Read every file as a syntax tree and run a short list of rules on commit. Run the full set, with coverage on changed lines, on the pull request. Block only what must never ship, and warn on the rest, because noise kills adoption.

The pages

Read in order, or jump in

FAQ

Questions people ask

What is shift-left testing for data engineering?

Catching problems in PySpark, Python and SQL code before it runs: lint, security and cost rules, unit tests on tiny DataFrames, a plan check on empty tables, and job-config rules, on commit and on the pull request.

Where should I start?

Shift left explained, then pure functions, then tests and coverage. The bundle page shows the tools and where the work stands.

Related

Where it connects

Data project review checks a project's documents; this guide checks its code. The open repo is shift-left-bundle on GitHub.