Home › Learn › Shift left › Shift left, explained

Shift left · page 1 of 13

Shift left: find problems before the code runs

Shift left means finding problems in PySpark, Python and SQL code before it runs: in the editor, on commit and on the pull request. What it can and can't catch.

↑ All of shift leftShift left, explainedHow software solved itSyntax treesPure functionsTests, coverage, complexityTesting data logic offlineClean codeSecurity and costWhat is missingThe bundleHow it is builtAll the checksOpen source and ours

In plain words

Move the checks to the left

Draw the life of a change from left to right: you write it, commit it, open a pull request, deploy it, it runs, and the bill arrives. A problem costs more the further right it is found. A cartesian join caught in the editor costs one minute. The same join found on the cluster costs a failed job, a bill and a morning of someone's time.

Shift left means moving as many checks as possible to the left end, where they need no cluster and no data: reading the code, running unit tests on tiny DataFrames, and checking job settings before deploy.

shift-left checks work here: no cluster, no dataEditorsecondsCommitunder 10 sPull requestunder 5 minPre-deployunder 1 minTest runhoursProductiona failed jobThe billmoney spentthe cost of finding a problem goes up this way
The dashed box is where shift-left checks run. Everything inside it is cheap and fast.

Be honest about it

What it can catch, and what it can't

Can catch before runningCan't catch without running
Code shapes that are always wrong or costly: collect in a loop, SELECT *, cross join, DELETE without WHEREData skew and real data volumes
Secrets in the code, libraries with known vulnerabilities, unsafe codeFlaky sources, late files, network failures
Logic bugs, through unit tests on tiny DataFramesWhat the cluster actually costs at run time
Untested and overly complex functionsProblems that only appear with production data
A bad query plan, from EXPLAIN on empty tablesPerformance at full scale
Costly job and cluster settingsWhether the business rule itself is right

So shift left is one part of testing and code review, not all of it. It removes the cheap problems early, so the expensive tests and the reviewers spend their time on the hard ones.

The one rule

Noise kills adoption

If a check cries wolf, developers stop reading all the checks. So at the start, only problems that must never ship block a change: security holes, leaked secrets and cartesian joins. Everything else warns. Existing code is recorded in a baseline, so only new problems fail. Rules that are often wrong are fixed or dropped. See how the software world solved it.

FAQ

Questions people ask

What is shift-left testing?

Moving checks to the earliest point in a change's life, the editor, the commit and the pull request, where they are fast and need no cluster or data. A problem found there costs minutes instead of a failed job and a bill.

Can static analysis replace running the pipeline?

No. It catches code shapes, secrets, vulnerable libraries, untested logic and bad plans. It can't see data skew, real volumes or run-time cost. Those need integration tests and monitoring.

Should every finding block a merge?

No. Block only what must never ship (security, secrets, cartesian joins), warn on the rest, and baseline existing code so only new problems fail.

Next

Where to go next

Next page: How the software world already solved this

↑ All of shift left · The repo on GitHub · Data project review