Home › Learn › Shift left › Open source and ours
Shift left · page 13 of 13
Open source we use, and what we added
The open-source tools behind the shift-left checks, with licences and what each finds, the tools considered and dropped, and what we built on top.
The split
57 checks come from open-source tools, 77 we wrote
The rule from the start was to reuse what the industry already trusts and build only what nobody ships. So most of the general checks, such as Python lint, security, secrets, vulnerable libraries and SQL style, are other people's rules that we switch on and configure. What we wrote is mostly specific to data work: cost in Spark code, plans and job config; whether logic is pure and tested; metric views; and what a data repository is missing.
Open source
The open-source tools behind the checks
| Tool | Licence | What it does here | Checks it finds | How it runs |
|---|---|---|---|---|
| Ruff | MIT | Python lint, clean code and security rules (most of Bandit), one fast tool | 33 | Runner |
| complexipy | MIT | cognitive complexity per function: how hard it is to read | 1 | Runner |
| vulture | MIT | dead code | 1 | Runner |
| pylint | GPL-2.0 | module length (on trial) | 1 | Not wired yet |
| jscpd | MIT | copied code in Python and SQL | 1 | Runner |
| SQLFluff | MIT | SQL lint and format, Databricks dialect | 2 | Runner |
| gitleaks | MIT | secrets in files and git history | 2 | Runner |
| pip-audit | Apache-2.0 | known vulnerabilities in Python libraries (fallback) | none yet | Not used by a check yet |
| osv-scanner | Apache-2.0 | known vulnerabilities from lockfiles (the OSV database) | 1 | Runner |
| nbstripout | MIT | strips notebook outputs on commit, so data samples don't leak | 1 | Runner |
| dbt-bouncer | MIT | rules over a dbt project's manifest, offline | 5 | Runner |
| dbt-score | MIT | scores dbt model metadata, offline | none yet | Not used by a check yet |
| dbt-project-evaluator | Apache-2.0 | dbt project best-practice rules (needs the warehouse, so trial only) | none yet | Not used by a check yet |
| Databricks EXPLAIN for materialized views | Databricks SQL | asks Databricks whether a materialized view can refresh incrementally, before deploy | 1 | Runner |
| SparkDoctor | Apache-2.0 | PySpark performance lint; only its mapped rules are used | 4 | Runner |
| pyspark-antipattern | MIT | PySpark lint; two of its rules adopted (join with no condition, inferSchema) | 1 | Runner |
| pip-licenses | MIT | lists library licences (on trial) | 1 | Not wired yet |
| coverage.py and diff-cover | Apache-2.0 | test coverage, measured on changed lines only | 1 | CI template |
| mutmut | BSD-3-Clause | mutation testing: changes the code and checks a test fails | 1 | Not wired yet |
| import-linter | BSD-2-Clause | architecture rule: the pure core never imports the IO code | 1 | Not wired yet |
| OpenSSF Scorecard | Apache-2.0 | scores a repository's security practices | 2 | Not wired yet |
| databricks bundle validate | Databricks CLI | validates Databricks bundle config | none yet | Not used by a check yet |
| sqlglot | MIT | SQL syntax tree with a Databricks dialect; our SQL and cost rules are written on it | the base of our SQL rules | Runner |
| Python's own ast | built in | Python syntax tree; our pattern rules, purity report and cost rules read it | the base of our Python rules | Runner |
| Spark's own planner | Apache-2.0 | EXPLAIN on local Spark gives the real plan of DataFrame code; our plan check reads it | the base of our plan check | Runner |
Two young PySpark linters from 2026 were tried against the sample before adoption. SparkDoctor is adopted for the rules that map to our checks; pyspark-antipattern for two rules only, because many of its others are opinions that hit our traps.
Used around the checks
| Tool | Licence | What it does here |
|---|---|---|
| pre-commit | MIT | runs the same hooks on the laptop and in CI from one file |
| pytest and pyspark.testing | built into Spark 3.5 | unit tests on tiny DataFrames; assertDataFrameEqual needs no extra library |
| Hypothesis | MPL-2.0 | property-based tests, the best fit for pure functions |
| syrupy | MIT | snapshot tests that pin an untested function's behaviour before a refactor |
| SARIF | open standard | findings shown on the pull request in GitHub or Azure DevOps, with no server to run |
Considered and not used
| Tool | Why not |
|---|---|
| SonarQube | needs a server and a database; pull-request analysis is in paid editions; knows nothing about Spark cost. Our SARIF output can be imported into it. |
| MegaLinter | an image of several gigabytes; we borrow its idea of one config for many tools |
| Bandit | Ruff's S rules cover most of it |
| Semgrep registry rules | their licence allows internal use only, so they can't ship in an open bundle; we write our own rules |
| detect-secrets, radon, wily | no recent releases |
| chispa | Spark 3.5 has DataFrame equality built in |
| cylint | every finding on the sample duplicated SparkDoctor or pyspark-antipattern |
Our delta
What we added
Rules and reports, by where they live:
| Part | What it does | Checks | How it runs |
|---|---|---|---|
| Opengrep (our rule) | pattern-rule engine; used for one rule of ours (credentials in config files) | 1 | Not wired yet |
| Our SQL rules (on sqlglot) | SQL syntax tree; we write our own rules on it | 4 | Runner |
| Our metric view checks | checks metric view YAML and parses its expressions | 7 | Runner |
| Our repo checks | what a repository is missing: tests, lockfile, pre-commit, CI setup | 9 | Runner |
| Our bundle config rules | Databricks bundle config rules: tags, timeouts, worker limits, auto-termination | 2 | Runner |
| Our test kit | standard test tables for known data patterns: SCD2, dedupe, incremental, late data | 1 | Test kit (pytest) |
| Our repo-review plugins | plugin host for repository checks; we write the data-repo plugins | 4 | repo-review plugin |
| Our cost rules | cost rules for SQL, Python, query plans and job config, from our earlier cost-analysis work | 23 | Runner |
| Our empty-table plan check | EXPLAIN on empty tables built from real schemas, on local Spark | 4 | Runner |
| Our purity report | pure, impure or unknown for every function, with the reason and a refactor hint | 6 | Runner |
| Our pattern rules | our own pattern rules over Python syntax trees and notebook text | 15 | Runner |
| Our test rules | rules over test files and coverage data | 1 | Runner |
And the parts that are not checks:
| Part | Why it matters |
|---|---|
| The catalog | 137 checks written once, with an id that names no product, the reason, the fix, the stage, the gate, the cost impact and links to data assurance obligations |
| The binding | maps each check to a tool and rule for one platform, so another platform is a new binding, not new checks |
| One runner and gate | calls every tool through an adapter, turns all output into one kind of finding, and decides block, warn or report |
| The baseline and allowed exceptions | an existing codebase isn't blocked by its past, and every exception carries a reason |
| Noise rules and the source trace | fourteen rules learned on real repositories, plus tracing where each value put into SQL comes from |
| The sample repo and answer key | planted problems, traps and a scoring script, so catch rates are measured |
| The SCD2 example and test kit | standard test tables for history tracking, de-duplication, incremental loads and late data |
| Test drafts and the health page | starter tests for pure functions with none, a pull-request summary and a repository health page with a trend |
| Templates | pre-commit, GitHub Actions, Azure Pipelines, a dev container, and Ruff profiles of 404 and 600 rules |
The cost rules came from earlier cost-analysis work and plug in as one adapter; nothing was rebuilt. Everything here is open: the repo on GitHub.
FAQ
Questions people ask
Which open-source tools does a shift-left setup for PySpark need?
Ruff for Python lint, clean code and security; SQLFluff and sqlglot for SQL; gitleaks for secrets; osv-scanner for vulnerable libraries; coverage.py and diff-cover for tests on changed lines; and a PySpark linter such as SparkDoctor for Spark-specific patterns.
What do the open-source tools not cover for data engineering?
Spark cost in code, plans and job config; whether logic is written as pure functions and tested; plan problems found on empty tables; metric views; and what a data repository is missing.
Next
Where to go next
↑ All of shift left · The repo on GitHub · Data project review