Home › Learn › Shift left › The bundle
Shift left · page 10 of 13
The shift-left bundle: design, tools and status
The shift-left bundle for PySpark and Databricks: spec, bindings and adapters, check families, tools with verdicts, why not SonarQube, git layout and status.
Where it stands
Status: the spec is written and the runner works
The plan, the reuse check and the eight starting decisions are done, and so is the spec: 137 checks in 13 families, with ids that name no product, each linked to the data assurance obligations it gives evidence for. They cover Python, PySpark, notebooks, SQL, dbt, materialized views and metric views. 20 block from day one, 85 warn and 32 only report. For Databricks, 81 of the 137 are found by rules that already exist, 54 we write, and 2 are on trial. The full list is in the repo's catalog index. Then a sample repo with planted problems and a hidden answer key, so catch rates are measured, not claimed. The repo is open: github.com/dataagentsai/shift-left-bundle.
| Tier | What | Status |
|---|---|---|
| 0 | Ground: reuse check, decisions, plan, open repo, this guide | done |
| 1 | Spec v0.1: 90 checks in 12 families, purity lists, gate policy, Databricks binding, a linter for the catalog | done |
| 1b | Spec 0.2: notebooks, dbt, materialized views, metric views, clean code on 404 Ruff rules, an SCD2 example with a test kit, CI and container templates | done |
| 2 | Proof material: a sample shop with 101 planted problems, 16 false-alarm traps and a hidden answer key. Ruff alone catches 17 of 101 with no false alarms; the rest is what the next tiers build | done |
| 3 | One runner over eight open-source tools, findings as SARIF on the pull request, block or warn, baseline for existing code. First measured run: 34 of 101 planted problems caught, no false alarms, under 8 seconds | done |
| 4 | Cost rules plugged in (our earlier cost-analysis rules, SparkDoctor after a trial of three young PySpark linters, our own materialized-view and job-config rules). Second measured run: 64 of 101 caught, still no false alarms | done |
| 5 | What we build: purity report, test-time purity guard, empty-table plan check, metric view and dbt checks, test drafts, a health page and pull-request summary. Fifth measured run: 88 of 101 caught; with our own pattern rules for the last nine checks, the sixth run catches 97 of 101, still no false alarms | done |
| 6 | A real repo: cut the noise, feed gaps back into the spec. First repo (13 Databricks notebooks): ten noise rules cut 19 blocking findings to 4, all four correct, and 276 findings to 172. A rule-by-rule comparison with Sonar added five PySpark checks (typeless null columns, window frames, join type, whole-row dedupe, long withColumn chains); the sample now has 106 planted problems and 102 are caught, still with no false alarms. Second repo (a streaming framework): four more noise rules cut 228 findings to 117, and the 3 blockers left are correct. Tracing each value put into SQL back to its source (a widget, the command line, a table) leaves one blocker on the first repo, and the message names the line the widget value came from. Two more checks followed: windows ordered with no partition, and copied code; the sample (108 planted) is caught 104 times | in progress |
| 7 | Teach and release: this guide kept current, a Claude skill, CI templates, version 0.1 | alongside |
| 8 | Azure binding | later |
Spec-driven
Spec, bindings, adapters
It follows the same shape as the other open specifications on this site. The catalog says what to check and why, and names no product. A binding per platform (Databricks first, then Azure) maps each check to a tool and its rule. The runner talks to every tool through one interface, so swapping a tool means writing one adapter, not changing the checks.
| Family | Asks | Starting gate |
|---|---|---|
| COST-CODE, COST-PLAN, COST-CONFIG | Does the code, the plan or the job config waste money? | warn; cartesian joins and config problems block |
| SEC, SECRET, DEP | Is it unsafe, leaking secrets, or using vulnerable libraries? | block |
| PURE, TEST | Is the logic pure and tested, and do the tests catch bugs? | report; coverage on changed lines blocks below 80% |
| CPLX, CLEAN, STYLE | Is it tangled, unclean or inconsistent? | warn; safe fixes automatic |
| SETUP | What is the repo missing? | report |
Deterministic first: every check is a tool or a script with a fixed answer. A language model is used only to explain a finding or draft a refactor, never to decide pass or fail.
Reuse
The tools, with verdicts
| Need | Tool | Verdict | Why |
|---|---|---|---|
| Runner | pre-commit | Pursue | Same configuration on the laptop and in CI |
| Python lint, clean code, security | Ruff | Pursue | One fast tool replaces flake8, isort, black, most of Bandit and many pylint rules |
| Custom pattern rules | Opengrep | Pursue | Open engine for our own rules |
| SQL | sqlglot, SQLFluff | Pursue | Syntax-tree rules and style |
| Purity | deal | Borrow | Generic side effects; the Spark part is ours |
| Tests and coverage | pytest, assertDataFrameEqual, coverage.py, diff-cover, Hypothesis, mutmut | Pursue | Standard |
| Complexity | Ruff C901, complexipy | Pursue | radon and wily are stale |
| Architecture rule | import-linter | Pursue | The core never imports the IO code |
| Secrets, vulnerabilities, infrastructure | gitleaks, osv-scanner, Checkov | Pursue | Standard and light |
| Gap report | repo-review, OpenSSF Scorecard | Pursue | Frameworks exist |
| Showing findings | SARIF on GitHub or Azure DevOps | Pursue | No server to run |
| Young PySpark linters | pyspark-antipattern, SparkDoctor | Trial | From 2026, one maintainer each; keep only what they add |
| Heavy platforms | SonarQube, MegaLinter, DefectDojo | Drop | A server or a multi-gigabyte image |
A common question
Why not SonarQube?
SonarQube needs a server and a database, pull-request analysis is in its paid editions, and it knows nothing about Spark cost or Databricks. What it gives, rules, a dashboard and "clean as you code", comes lighter here: Ruff and Opengrep for rules, SARIF findings on the pull request for the dashboard, and diff-cover with a baseline for clean as you code. Because the output is SARIF, a team that already runs SonarQube can import the findings.
Git
How the repo is laid out, and how a team uses it
shift-left-bundle/ .pre-commit-hooks.yaml teams install the bundle from here (a few lines in their config) catalog/checks/*.yaml the spec, versioned on its own bindings/databricks.yaml bindings/azure.yaml (the only place product names appear) rules/ Opengrep, SQLFluff, Ruff and import-linter configuration src/shiftleft/ runner: interfaces, adapters, purity, empty-table harness, report plugins/repo_review/ the gap-report checks templates/ pre-commit config, GitHub Actions and Azure DevOps pipelines examples/sample-project/ planted problems; tests/answer-key.yaml
Short-lived branches merge into a protected main through pull requests, with the bundle's pull-request job as a required check. Teams pin a version tag of the bundle, so a new rule never breaks them by surprise.
FAQ
Questions people ask
Do I need SonarQube for a PySpark project?
No. Ruff, Opengrep, SQLFluff, gitleaks, pip-audit and coverage tools cover the same ground with no server, and SARIF output shows findings on the pull request. SonarQube also has no Spark cost rules.
What is spec-driven about this bundle?
The checks are written once, with ids and reasons and no product names. Each platform gets a binding that maps checks to tools, so adding Azure or another platform is a new binding, not new checks.
Can I use it today?
Yes, as a preview: `shiftleft scan . --stage pr`, `shiftleft purity .` and `shiftleft draft-tests .` work today, as does the pre-commit hook. Version 0.1 follows a run on a real repo. Follow the GitHub repo.
Next
Where to go next
Next page: How the shift-left bundle is built, and how we know it works
↑ All of shift left · The repo on GitHub · Data project review