Home › Learn › Shift left › Open source and ours

Shift left · page 13 of 13

Open source we use, and what we added

The open-source tools behind the shift-left checks, with licences and what each finds, the tools considered and dropped, and what we built on top.

↑ All of shift leftShift left, explainedHow software solved itSyntax treesPure functionsTests, coverage, complexityTesting data logic offlineClean codeSecurity and costWhat is missingThe bundleHow it is builtAll the checksOpen source and ours

The split

57 checks come from open-source tools, 77 we wrote

The rule from the start was to reuse what the industry already trusts and build only what nobody ships. So most of the general checks, such as Python lint, security, secrets, vulnerable libraries and SQL style, are other people's rules that we switch on and configure. What we wrote is mostly specific to data work: cost in Spark code, plans and job config; whether logic is pure and tested; metric views; and what a data repository is missing.

Open-source tool's own ruleWritten by usOn trialThe platform's own commandCLEAN248COST-CODE520COST-CONFIG91COST-PLAN6CPLX611DEP141PURE16SEC73SECRET21SEM7SETUP37STYLE4TEST45
Each bar is one family of checks. Blue is an open-source tool's own rule; green is ours.

Open source

The open-source tools behind the checks

ToolLicenceWhat it does hereChecks it findsHow it runs
RuffMITPython lint, clean code and security rules (most of Bandit), one fast tool33Runner
complexipyMITcognitive complexity per function: how hard it is to read1Runner
vultureMITdead code1Runner
pylintGPL-2.0module length (on trial)1Not wired yet
jscpdMITcopied code in Python and SQL1Runner
SQLFluffMITSQL lint and format, Databricks dialect2Runner
gitleaksMITsecrets in files and git history2Runner
pip-auditApache-2.0known vulnerabilities in Python libraries (fallback)none yetNot used by a check yet
osv-scannerApache-2.0known vulnerabilities from lockfiles (the OSV database)1Runner
nbstripoutMITstrips notebook outputs on commit, so data samples don't leak1Runner
dbt-bouncerMITrules over a dbt project's manifest, offline5Runner
dbt-scoreMITscores dbt model metadata, offlinenone yetNot used by a check yet
dbt-project-evaluatorApache-2.0dbt project best-practice rules (needs the warehouse, so trial only)none yetNot used by a check yet
Databricks EXPLAIN for materialized viewsDatabricks SQLasks Databricks whether a materialized view can refresh incrementally, before deploy1Runner
SparkDoctorApache-2.0PySpark performance lint; only its mapped rules are used4Runner
pyspark-antipatternMITPySpark lint; two of its rules adopted (join with no condition, inferSchema)1Runner
pip-licensesMITlists library licences (on trial)1Not wired yet
coverage.py and diff-coverApache-2.0test coverage, measured on changed lines only1CI template
mutmutBSD-3-Clausemutation testing: changes the code and checks a test fails1Not wired yet
import-linterBSD-2-Clausearchitecture rule: the pure core never imports the IO code1Not wired yet
OpenSSF ScorecardApache-2.0scores a repository's security practices2Not wired yet
databricks bundle validateDatabricks CLIvalidates Databricks bundle confignone yetNot used by a check yet
sqlglotMITSQL syntax tree with a Databricks dialect; our SQL and cost rules are written on itthe base of our SQL rulesRunner
Python's own astbuilt inPython syntax tree; our pattern rules, purity report and cost rules read itthe base of our Python rulesRunner
Spark's own plannerApache-2.0EXPLAIN on local Spark gives the real plan of DataFrame code; our plan check reads itthe base of our plan checkRunner

Two young PySpark linters from 2026 were tried against the sample before adoption. SparkDoctor is adopted for the rules that map to our checks; pyspark-antipattern for two rules only, because many of its others are opinions that hit our traps.

Used around the checks

ToolLicenceWhat it does here
pre-commitMITruns the same hooks on the laptop and in CI from one file
pytest and pyspark.testingbuilt into Spark 3.5unit tests on tiny DataFrames; assertDataFrameEqual needs no extra library
HypothesisMPL-2.0property-based tests, the best fit for pure functions
syrupyMITsnapshot tests that pin an untested function's behaviour before a refactor
SARIFopen standardfindings shown on the pull request in GitHub or Azure DevOps, with no server to run

Considered and not used

ToolWhy not
SonarQubeneeds a server and a database; pull-request analysis is in paid editions; knows nothing about Spark cost. Our SARIF output can be imported into it.
MegaLinteran image of several gigabytes; we borrow its idea of one config for many tools
BanditRuff's S rules cover most of it
Semgrep registry rulestheir licence allows internal use only, so they can't ship in an open bundle; we write our own rules
detect-secrets, radon, wilyno recent releases
chispaSpark 3.5 has DataFrame equality built in
cylintevery finding on the sample duplicated SparkDoctor or pyspark-antipattern

Our delta

What we added

Rules and reports, by where they live:

PartWhat it doesChecksHow it runs
Opengrep (our rule)pattern-rule engine; used for one rule of ours (credentials in config files)1Not wired yet
Our SQL rules (on sqlglot)SQL syntax tree; we write our own rules on it4Runner
Our metric view checkschecks metric view YAML and parses its expressions7Runner
Our repo checkswhat a repository is missing: tests, lockfile, pre-commit, CI setup9Runner
Our bundle config rulesDatabricks bundle config rules: tags, timeouts, worker limits, auto-termination2Runner
Our test kitstandard test tables for known data patterns: SCD2, dedupe, incremental, late data1Test kit (pytest)
Our repo-review pluginsplugin host for repository checks; we write the data-repo plugins4repo-review plugin
Our cost rulescost rules for SQL, Python, query plans and job config, from our earlier cost-analysis work23Runner
Our empty-table plan checkEXPLAIN on empty tables built from real schemas, on local Spark4Runner
Our purity reportpure, impure or unknown for every function, with the reason and a refactor hint6Runner
Our pattern rulesour own pattern rules over Python syntax trees and notebook text15Runner
Our test rulesrules over test files and coverage data1Runner

And the parts that are not checks:

PartWhy it matters
The catalog137 checks written once, with an id that names no product, the reason, the fix, the stage, the gate, the cost impact and links to data assurance obligations
The bindingmaps each check to a tool and rule for one platform, so another platform is a new binding, not new checks
One runner and gatecalls every tool through an adapter, turns all output into one kind of finding, and decides block, warn or report
The baseline and allowed exceptionsan existing codebase isn't blocked by its past, and every exception carries a reason
Noise rules and the source tracefourteen rules learned on real repositories, plus tracing where each value put into SQL comes from
The sample repo and answer keyplanted problems, traps and a scoring script, so catch rates are measured
The SCD2 example and test kitstandard test tables for history tracking, de-duplication, incremental loads and late data
Test drafts and the health pagestarter tests for pure functions with none, a pull-request summary and a repository health page with a trend
Templatespre-commit, GitHub Actions, Azure Pipelines, a dev container, and Ruff profiles of 404 and 600 rules

The cost rules came from earlier cost-analysis work and plug in as one adapter; nothing was rebuilt. Everything here is open: the repo on GitHub.

FAQ

Questions people ask

Which open-source tools does a shift-left setup for PySpark need?

Ruff for Python lint, clean code and security; SQLFluff and sqlglot for SQL; gitleaks for secrets; osv-scanner for vulnerable libraries; coverage.py and diff-cover for tests on changed lines; and a PySpark linter such as SparkDoctor for Spark-specific patterns.

What do the open-source tools not cover for data engineering?

Spark cost in code, plans and job config; whether logic is written as pure functions and tested; plan problems found on empty tables; metric views; and what a data repository is missing.

Next

Where to go next

↑ All of shift left · The repo on GitHub · Data project review