How to catch problems in PySpark, Python and Databricks SQL before the code runs: cost, security, vulnerable libraries, untested logic, tangled functions, and what a repo is missing. Built from open-source tools the software world already trusts, with no heavy servers.
The picture
Find it on the left, where it is cheap
Shift-left checks live in the dashed box: no cluster, no data, seconds to minutes.
The short answer: make most of the logic pure functions (DataFrame in, DataFrame out) so it can be unit tested in seconds. Read every file as a syntax tree and run a short list of rules on commit. Run the full set, with coverage on changed lines, on the pull request. Block only what must never ship, and warn on the rest, because noise kills adoption.
Catching problems in PySpark, Python and SQL code before it runs: lint, security and cost rules, unit tests on tiny DataFrames, a plan check on empty tables, and job-config rules, on commit and on the pull request.
Where should I start?
Shift left explained, then pure functions, then tests and coverage. The bundle page shows the tools and where the work stands.