Home › Learn › Shift left › The bundle

Shift left · page 10 of 13

The shift-left bundle: design, tools and status

The shift-left bundle for PySpark and Databricks: spec, bindings and adapters, check families, tools with verdicts, why not SonarQube, git layout and status.

↑ All of shift leftShift left, explainedHow software solved itSyntax treesPure functionsTests, coverage, complexityTesting data logic offlineClean codeSecurity and costWhat is missingThe bundleHow it is builtAll the checksOpen source and ours

Where it stands

Status: the spec is written and the runner works

The plan, the reuse check and the eight starting decisions are done, and so is the spec: 137 checks in 13 families, with ids that name no product, each linked to the data assurance obligations it gives evidence for. They cover Python, PySpark, notebooks, SQL, dbt, materialized views and metric views. 20 block from day one, 85 warn and 32 only report. For Databricks, 81 of the 137 are found by rules that already exist, 54 we write, and 2 are on trial. The full list is in the repo's catalog index. Then a sample repo with planted problems and a hidden answer key, so catch rates are measured, not claimed. The repo is open: github.com/dataagentsai/shift-left-bundle.

TierWhatStatus
0Ground: reuse check, decisions, plan, open repo, this guidedone
1Spec v0.1: 90 checks in 12 families, purity lists, gate policy, Databricks binding, a linter for the catalogdone
1bSpec 0.2: notebooks, dbt, materialized views, metric views, clean code on 404 Ruff rules, an SCD2 example with a test kit, CI and container templatesdone
2Proof material: a sample shop with 101 planted problems, 16 false-alarm traps and a hidden answer key. Ruff alone catches 17 of 101 with no false alarms; the rest is what the next tiers builddone
3One runner over eight open-source tools, findings as SARIF on the pull request, block or warn, baseline for existing code. First measured run: 34 of 101 planted problems caught, no false alarms, under 8 secondsdone
4Cost rules plugged in (our earlier cost-analysis rules, SparkDoctor after a trial of three young PySpark linters, our own materialized-view and job-config rules). Second measured run: 64 of 101 caught, still no false alarmsdone
5What we build: purity report, test-time purity guard, empty-table plan check, metric view and dbt checks, test drafts, a health page and pull-request summary. Fifth measured run: 88 of 101 caught; with our own pattern rules for the last nine checks, the sixth run catches 97 of 101, still no false alarmsdone
6A real repo: cut the noise, feed gaps back into the spec. First repo (13 Databricks notebooks): ten noise rules cut 19 blocking findings to 4, all four correct, and 276 findings to 172. A rule-by-rule comparison with Sonar added five PySpark checks (typeless null columns, window frames, join type, whole-row dedupe, long withColumn chains); the sample now has 106 planted problems and 102 are caught, still with no false alarms. Second repo (a streaming framework): four more noise rules cut 228 findings to 117, and the 3 blockers left are correct. Tracing each value put into SQL back to its source (a widget, the command line, a table) leaves one blocker on the first repo, and the message names the line the widget value came from. Two more checks followed: windows ordered with no partition, and copied code; the sample (108 planted) is caught 104 timesin progress
7Teach and release: this guide kept current, a Claude skill, CI templates, version 0.1alongside
8Azure bindinglater

Spec-driven

Spec, bindings, adapters

It follows the same shape as the other open specifications on this site. The catalog says what to check and why, and names no product. A binding per platform (Databricks first, then Azure) maps each check to a tool and its rule. The runner talks to every tool through one interface, so swapping a tool means writing one adapter, not changing the checks.

FamilyAsksStarting gate
COST-CODE, COST-PLAN, COST-CONFIGDoes the code, the plan or the job config waste money?warn; cartesian joins and config problems block
SEC, SECRET, DEPIs it unsafe, leaking secrets, or using vulnerable libraries?block
PURE, TESTIs the logic pure and tested, and do the tests catch bugs?report; coverage on changed lines blocks below 80%
CPLX, CLEAN, STYLEIs it tangled, unclean or inconsistent?warn; safe fixes automatic
SETUPWhat is the repo missing?report

Deterministic first: every check is a tool or a script with a fixed answer. A language model is used only to explain a finding or draft a refactor, never to decide pass or fail.

Reuse

The tools, with verdicts

NeedToolVerdictWhy
Runnerpre-commitPursueSame configuration on the laptop and in CI
Python lint, clean code, securityRuffPursueOne fast tool replaces flake8, isort, black, most of Bandit and many pylint rules
Custom pattern rulesOpengrepPursueOpen engine for our own rules
SQLsqlglot, SQLFluffPursueSyntax-tree rules and style
PuritydealBorrowGeneric side effects; the Spark part is ours
Tests and coveragepytest, assertDataFrameEqual, coverage.py, diff-cover, Hypothesis, mutmutPursueStandard
ComplexityRuff C901, complexipyPursueradon and wily are stale
Architecture ruleimport-linterPursueThe core never imports the IO code
Secrets, vulnerabilities, infrastructuregitleaks, osv-scanner, CheckovPursueStandard and light
Gap reportrepo-review, OpenSSF ScorecardPursueFrameworks exist
Showing findingsSARIF on GitHub or Azure DevOpsPursueNo server to run
Young PySpark linterspyspark-antipattern, SparkDoctorTrialFrom 2026, one maintainer each; keep only what they add
Heavy platformsSonarQube, MegaLinter, DefectDojoDropA server or a multi-gigabyte image

A common question

Why not SonarQube?

SonarQube needs a server and a database, pull-request analysis is in its paid editions, and it knows nothing about Spark cost or Databricks. What it gives, rules, a dashboard and "clean as you code", comes lighter here: Ruff and Opengrep for rules, SARIF findings on the pull request for the dashboard, and diff-cover with a baseline for clean as you code. Because the output is SARIF, a team that already runs SonarQube can import the findings.

Git

How the repo is laid out, and how a team uses it

shift-left-bundle/
  .pre-commit-hooks.yaml     teams install the bundle from here (a few lines in their config)
  catalog/checks/*.yaml      the spec, versioned on its own
  bindings/databricks.yaml   bindings/azure.yaml   (the only place product names appear)
  rules/                     Opengrep, SQLFluff, Ruff and import-linter configuration
  src/shiftleft/             runner: interfaces, adapters, purity, empty-table harness, report
  plugins/repo_review/       the gap-report checks
  templates/                 pre-commit config, GitHub Actions and Azure DevOps pipelines
  examples/sample-project/   planted problems; tests/answer-key.yaml
On commitunder 10 slint, clean code, security, secrets, SQL and cost rules on changed filesPull requestunder 5 min+ unit tests, coverage on changed lines, purity, complexity, vulnerable libraries, plan checkNightlyno limitmutation testing, repo security score, gap report, trendsBefore deployunder 1 minjob and cluster config rules, bundle validation
A team's repo runs the fast checks on commit and the full set on the pull request.

Short-lived branches merge into a protected main through pull requests, with the bundle's pull-request job as a required check. Teams pin a version tag of the bundle, so a new rule never breaks them by surprise.

FAQ

Questions people ask

Do I need SonarQube for a PySpark project?

No. Ruff, Opengrep, SQLFluff, gitleaks, pip-audit and coverage tools cover the same ground with no server, and SARIF output shows findings on the pull request. SonarQube also has no Spark cost rules.

What is spec-driven about this bundle?

The checks are written once, with ids and reasons and no product names. Each platform gets a binding that maps checks to tools, so adding Azure or another platform is a new binding, not new checks.

Can I use it today?

Yes, as a preview: `shiftleft scan . --stage pr`, `shiftleft purity .` and `shiftleft draft-tests .` work today, as does the pre-commit hook. Version 0.1 follows a run on a real repo. Follow the GitHub repo.

Next

Where to go next

Next page: How the shift-left bundle is built, and how we know it works

↑ All of shift left · The repo on GitHub · Data project review