Home › Learn › Shift left › Security and cost

Shift left · page 8 of 13

Security, secrets, vulnerable libraries and cost

Static checks for PySpark and Databricks SQL: SQL injection, unsafe calls, secrets, vulnerable libraries, and cost problems in code, SQL, plans and job config.

↑ All of shift leftShift left, explainedHow software solved itSyntax treesPure functionsTests, coverage, complexityTesting data logic offlineClean codeSecurity and costWhat is missingThe bundleHow it is builtAll the checksOpen source and ours

Security

Unsafe code, secrets and vulnerable libraries

ProblemExampleToolGate
SQL built from stringsspark.sql(f"... WHERE country = '{{country}}'")Ruff S608; Opengrep rulesblock
Unsafe callseval, pickle.loads, SSL verification turned off, subprocess with shell=TrueRuff S rules (from Bandit)block
Secret printed to a logprint(dbutils.secrets.get(...))Opengrep ruleblock
Secret in the repo or its historya token, key or connection stringgitleaksblock
Library with a known vulnerabilityan old requests or urllib3osv-scanner (pip-audit as fallback)block on high
Unpinned libraries, no lockfilepyspark>=3the gap reportwarn
Infrastructure misconfigurationopen storage, no encryption in TerraformCheckovblock
# Unsafe: the value becomes part of the SQL text
spark.sql(f"SELECT * FROM orders WHERE country = '{country}'")

# Safe: a named parameter (Spark 3.4 and later), and only the columns needed
spark.sql("SELECT order_id, amount FROM orders WHERE country = :country", args={"country": country})

Opengrep is the open-source fork of Semgrep's engine, used here for our own pattern rules. Semgrep's ready-made registry rules have a licence that limits them to internal use, so we write our own.

Cost

Cost problems you can see before running

Python codeloops, collect,UDFs, toPandasSQL textSELECT *, cross join,no partition filterQuery planEXPLAIN onempty tablesJob configauto-stop, clustertype, tagsCost findings
Four places to look. All of them are read without a cluster.
WhereExamplesGate
Python codecollect() or toPandas() without a limit, a Spark call inside a loop, a Python UDF where a built-in function exists, withColumn in a loop, count() to test for emptiness, cache with no unpersistwarn, then block once baselined
SQL textSELECT *, cross join, a function on a partition column, MERGE without a partition predicate, ORDER BY with no LIMIT, the same table scanned several timeswarn; cross join blocks
Query plan on empty tablescartesian product, nested-loop join, Python UDF evaluation, a scan with no filter pushed down, too many shufflescartesian blocks
Job and cluster configno auto-stop, an always-on cluster for a scheduled job, no cost tags, no limit on workers, continuous mode where triggered is enoughblock

These cost rules already exist from our earlier work on cost analysis, built on sqlglot and Python's ast: about 35 SQL rules, 11 Python rules, plan rules and job-config rules. They plug into the bundle as the cost adapter, so nothing is rebuilt. Younger open-source PySpark linters from 2026 (pyspark-antipattern, SparkDoctor) are tried against the sample repo, and only rules they add are kept.

FAQ

Questions people ask

How do I prevent SQL injection in spark.sql?

Don't build SQL from f-strings. Use named parameters, spark.sql(query, args={...}), available since Spark 3.4. Ruff's S608 rule flags SQL built from strings.

Which tool finds secrets in a git repository?

gitleaks scans files and git history for tokens, keys and connection strings. It is a single binary and runs in pre-commit and CI.

Can you find Spark cost problems without running the job?

Many of them: costly code shapes in Python and SQL, a cartesian product or Python UDF in the plan from EXPLAIN on empty tables, and costly job or cluster settings. Skew and real volumes still need a run.

Next

Where to go next

Next page: The gap report: what a repo is missing

↑ All of shift left · The repo on GitHub · Data project review