Home › Learn › Shift left › How it is built
Shift left · page 11 of 13
How the shift-left bundle is built, and how we know it works
How the shift-left bundle is built: a catalog of checks, a platform binding, one runner with adapters, gates, a baseline, noise rules and reports, and how its catch rate is measured.
The idea
Write each check once, run it with whatever tool finds it best
Shift left means finding a problem while the code is still on a laptop or in a pull request, not after it has run on a cluster. The bundle does that with three parts that are kept apart on purpose. The catalog says what to check and why, in plain words, with no product names. The binding says, for one platform, which tool and which of its rules finds each check. The runner calls the tools, turns everything they say into one kind of finding, and decides what blocks a merge.
Keeping them apart pays off twice. A tool can be swapped by writing one adapter (a small piece of code that calls the tool and translates its output), without touching the checks. And the checks stay the same on another platform: only the binding changes.
When checks run
Four stages, each with a time budget
Each check names the first stage where it runs, and it runs again at every later stage. Fast checks run on commit; slow ones, such as mutation testing (changing the code on purpose to see whether a test notices), run nightly.
| Stage | Time budget | What it looks at | Checks that start here |
|---|---|---|---|
| On commit | under 10 seconds | changed files only | 79 |
| Pull request | under 5 minutes | the whole change; the merge waits until it passes | 33 |
| Nightly | no limit | the whole repository and its history; slow checks live here | 15 |
| Before deploy | under 1 minute | job, compute and deployment configuration | 10 |
Block or warn
How a finding becomes a block
Each check has a starting gate, set in the catalog: block (20 checks) means the change can't merge or deploy until the finding is fixed or accepted; warn (85) shows on the pull request, on the changed line, without stopping the merge; report (32) only lists it, for counts and trends. A team can lower or raise any gate in its own config.
A check moves from warn to block only after at least four weeks as a warning, and only if developers marked no more than one in ten of its findings as wrong. Noise is the main reason teams switch checks off, so the bundle starts gentle.
Two ways keep an existing codebase from being blocked by its past. The baseline records the findings that existed when the bundle was adopted; they never block, only new ones do. Each is fingerprinted by the check id, the file and a hash of the code at the finding with whitespace removed, so moving code around doesn't make an old finding look new. And a single line can be allowed with a reason: a comment shiftleft: allow SL-SEC-01 -- table name from config on the line or the line above. A marker with no reason is ignored.
Noise rules run before the gate. Each one looks at a finding and the code around it and either drops it (the tool is wrong there) or lowers it to report (right, but not worth stopping a pull request). Every drop is counted in the run notes, never hidden. For SQL built from strings, a small source trace follows each value put into the SQL back to where it came from: a widget, the command line, a table or config. Untrusted input stays a block and the message names its source; config only reports.
How it is proven
A sample repo with planted problems and a hidden answer key
Claims about catch rates are easy to make, so the bundle measures them. A sample project, a small shop with PySpark, SQL, notebooks, dbt and bundle config, has problems planted on purpose and a hidden answer key listing each one. It also has traps: code that looks wrong but is fine, to catch rules that fire too eagerly. A scoring script compares each run's findings with the key.
Then real repositories, to cut the noise. On the first, 13 Databricks notebooks, ten noise rules cut 276 findings to 172 and the blocking ones from 19 to 4, all four correct; tracing where SQL values come from then left one. On the second, a streaming framework, four more noise rules cut 228 findings to 117, and the 3 blockers left are correct. A rule-by-rule comparison with Sonar's Python and PySpark rules added five PySpark checks, such as a window ordered with no explicit frame and a join with no join type.
Where it stands
The work, tier by tier
| Tier | What | Status |
|---|---|---|
| 0 | Ground: the reuse check, the starting decisions, the plan, the open repo and this guide | done |
| 1 and 1b | The spec: now 137 checks in 13 families, with notebooks, dbt, materialized views and metric views; an SCD2 example with a test kit; CI and container templates | done |
| 2 | The sample repo with planted problems, traps and the hidden answer key | done |
| 3 | One runner over the open-source tools, SARIF on the pull request, block or warn, the baseline | done |
| 4 | The cost rules plugged in, and a trial of the young PySpark linters | done |
| 5 | What nobody ships: the purity report, the empty-table plan check, metric view and dbt checks, test drafts, our own pattern rules | done |
| 6 | Real repositories: cut the noise and feed gaps back into the spec | in progress |
| 7 | Teach and release: this guide, a Claude skill, version 0.1 | alongside |
| 8 | An Azure binding | later |
The commands today: shiftleft scan . --stage pr (with --md, --sarif, --summary and --html for the reports), shiftleft baseline ., shiftleft purity . and shiftleft draft-tests .. The repo layout and the git flow are on The bundle; the full list of checks is on All the checks.
FAQ
Questions people ask
How do you stop a linter from blocking an old codebase?
Record a baseline when you adopt it. Findings in the baseline never block; only new ones do. Fingerprint them by check, file and a whitespace-free hash of the code, not by line number, so moving code doesn't create new findings.
How do you know a set of code checks actually works?
Plant problems in a sample repo with a hidden answer key and score each run. Add traps, code that looks wrong but is fine, so rules that fire too eagerly show up as false alarms.
Why keep the checks separate from the tools?
So a tool can be swapped by writing one adapter, and the same checks can be bound to another platform. The checks name no product; only the binding does.
Next
Where to go next
Next page: Every shift-left check, filterable
↑ All of shift left · The repo on GitHub · Data project review