Home › Learn › Shift left › Syntax trees
Shift left · page 3 of 13
Every language already has a syntax tree
Linters read code as a syntax tree. The parsers for Python (ast, LibCST), Databricks SQL (sqlglot, SQLFluff), PySpark (the Spark plan) and Databricks config.
In plain words
A linter reads a tree, not text
Searching code as plain text breaks easily: a comment or a string can look like a problem. A proper linter first parses the code into a syntax tree, where every call, join, filter and loop is a node. Rules then walk the tree: "if a collect() node sits inside a for node, report it."
Layer by layer
The parser for each layer
| Layer | What it becomes | Parser | Also used by |
|---|---|---|---|
| Python | Python syntax tree | ast (built into Python); LibCST to rewrite code safely; astroid to know what a name refers to | Most Python linters; Databricks UCX uses astroid |
| SQL, Databricks dialect | SQL syntax tree, with column lineage | sqlglot (MIT licence, very active) | Databricks UCX and Lakebridge |
| SQL style | SQL parse tree | SQLFluff, with a databricks dialect | dbt projects |
| PySpark DataFrame code | Spark's own plan (Catalyst), from EXPLAIN on local Spark with empty tables that have the real columns | Spark itself | Spark's optimiser |
| Databricks jobs and clusters | Bundle configuration (YAML) | the bundle schema and databricks bundle validate | Databricks CLI |
| Azure Data Factory | Pipeline JSON | JSON | Azure tooling |
The PySpark trick
The plan is PySpark's real syntax tree
PySpark code is Python, so the Python tree sees df.join(other) but not what Spark will do with it. Spark's plan does. Build each input as an empty DataFrame with the real columns, run the transformation, and call EXPLAIN. That takes a few seconds on a laptop, reads no data, and shows a cartesian product, Python UDF evaluation or a scan with no filter pushed down.
orders = spark.createDataFrame([], "order_id INT, customer_id INT, country STRING")
customers = spark.createDataFrame([], "customer_id INT, name STRING")
plan = orders.join(customers).filter("country = 'IN'") # no join key: a cartesian product
plan.explain() # the plan names the join type; a rule reports a cartesian product or nested-loop joinChecked on Spark 3.5.1: with empty inputs, the physical plan shows CartesianProduct, without reading any data.
It can't see sizes, so it can't judge broadcast choices or skew. Those need statistics or a real run.
FAQ
Questions people ask
What is an abstract syntax tree?
The structure of code after parsing: each call, condition, join and loop becomes a node in a tree. Linters walk the tree, which is far more reliable than searching the text.
Is there a SQL parser for Databricks SQL?
Yes. sqlglot parses the Databricks dialect into a syntax tree and is used by Databricks' own UCX and Lakebridge tools. SQLFluff also has a databricks dialect for style rules.
How can I check a PySpark plan without data?
Create empty DataFrames with the real schemas, run the transformation and call explain(). The physical plan shows join types, Python UDF evaluation and pushed filters in seconds.
Next
Where to go next
Next page: Pure functions in PySpark: possible, and how we detect them
↑ All of shift left · The repo on GitHub · Data project review