Home › Learn › Shift left › Security and cost
Shift left · page 8 of 13
Security, secrets, vulnerable libraries and cost
Static checks for PySpark and Databricks SQL: SQL injection, unsafe calls, secrets, vulnerable libraries, and cost problems in code, SQL, plans and job config.
Security
Unsafe code, secrets and vulnerable libraries
| Problem | Example | Tool | Gate |
|---|---|---|---|
| SQL built from strings | spark.sql(f"... WHERE country = '{{country}}'") | Ruff S608; Opengrep rules | block |
| Unsafe calls | eval, pickle.loads, SSL verification turned off, subprocess with shell=True | Ruff S rules (from Bandit) | block |
| Secret printed to a log | print(dbutils.secrets.get(...)) | Opengrep rule | block |
| Secret in the repo or its history | a token, key or connection string | gitleaks | block |
| Library with a known vulnerability | an old requests or urllib3 | osv-scanner (pip-audit as fallback) | block on high |
| Unpinned libraries, no lockfile | pyspark>=3 | the gap report | warn |
| Infrastructure misconfiguration | open storage, no encryption in Terraform | Checkov | block |
# Unsafe: the value becomes part of the SQL text
spark.sql(f"SELECT * FROM orders WHERE country = '{country}'")
# Safe: a named parameter (Spark 3.4 and later), and only the columns needed
spark.sql("SELECT order_id, amount FROM orders WHERE country = :country", args={"country": country})
Opengrep is the open-source fork of Semgrep's engine, used here for our own pattern rules. Semgrep's ready-made registry rules have a licence that limits them to internal use, so we write our own.
Cost
Cost problems you can see before running
| Where | Examples | Gate |
|---|---|---|
| Python code | collect() or toPandas() without a limit, a Spark call inside a loop, a Python UDF where a built-in function exists, withColumn in a loop, count() to test for emptiness, cache with no unpersist | warn, then block once baselined |
| SQL text | SELECT *, cross join, a function on a partition column, MERGE without a partition predicate, ORDER BY with no LIMIT, the same table scanned several times | warn; cross join blocks |
| Query plan on empty tables | cartesian product, nested-loop join, Python UDF evaluation, a scan with no filter pushed down, too many shuffles | cartesian blocks |
| Job and cluster config | no auto-stop, an always-on cluster for a scheduled job, no cost tags, no limit on workers, continuous mode where triggered is enough | block |
These cost rules already exist from our earlier work on cost analysis, built on sqlglot and Python's ast: about 35 SQL rules, 11 Python rules, plan rules and job-config rules. They plug into the bundle as the cost adapter, so nothing is rebuilt. Younger open-source PySpark linters from 2026 (pyspark-antipattern, SparkDoctor) are tried against the sample repo, and only rules they add are kept.
FAQ
Questions people ask
How do I prevent SQL injection in spark.sql?
Don't build SQL from f-strings. Use named parameters, spark.sql(query, args={...}), available since Spark 3.4. Ruff's S608 rule flags SQL built from strings.
Which tool finds secrets in a git repository?
gitleaks scans files and git history for tokens, keys and connection strings. It is a single binary and runs in pre-commit and CI.
Can you find Spark cost problems without running the job?
Many of them: costly code shapes in Python and SQL, a cartesian product or Python UDF in the plan from EXPLAIN on empty tables, and costly job or cluster settings. Skew and real volumes still need a run.
Next
Where to go next
Next page: The gap report: what a repo is missing
↑ All of shift left · The repo on GitHub · Data project review