Home › Learn › Shift left › Syntax trees

Shift left · page 3 of 13

Every language already has a syntax tree

Linters read code as a syntax tree. The parsers for Python (ast, LibCST), Databricks SQL (sqlglot, SQLFluff), PySpark (the Spark plan) and Databricks config.

↑ All of shift leftShift left, explainedHow software solved itSyntax treesPure functionsTests, coverage, complexityTesting data logic offlineClean codeSecurity and costWhat is missingThe bundleHow it is builtAll the checksOpen source and ours

In plain words

A linter reads a tree, not text

Searching code as plain text breaks easily: a comment or a string can look like a problem. A proper linter first parses the code into a syntax tree, where every call, join, filter and loop is a node. Rules then walk the tree: "if a collect() node sits inside a for node, report it."

Your code.py, .sql,notebook, YAMLparseSyntax treenodes: call, join,filter, loopwalkRulesif node looks like X,report itFindingfile, line, id,why, fixPull requestshown on thechanged linethe parser comes off the shelf; we only write the rules nobody else has
Parse, walk, report. Every layer of a data project has a mature parser, so we only write rules.

Layer by layer

The parser for each layer

LayerWhat it becomesParserAlso used by
PythonPython syntax treeast (built into Python); LibCST to rewrite code safely; astroid to know what a name refers toMost Python linters; Databricks UCX uses astroid
SQL, Databricks dialectSQL syntax tree, with column lineagesqlglot (MIT licence, very active)Databricks UCX and Lakebridge
SQL styleSQL parse treeSQLFluff, with a databricks dialectdbt projects
PySpark DataFrame codeSpark's own plan (Catalyst), from EXPLAIN on local Spark with empty tables that have the real columnsSpark itselfSpark's optimiser
Databricks jobs and clustersBundle configuration (YAML)the bundle schema and databricks bundle validateDatabricks CLI
Azure Data FactoryPipeline JSONJSONAzure tooling

The PySpark trick

The plan is PySpark's real syntax tree

PySpark code is Python, so the Python tree sees df.join(other) but not what Spark will do with it. Spark's plan does. Build each input as an empty DataFrame with the real columns, run the transformation, and call EXPLAIN. That takes a few seconds on a laptop, reads no data, and shows a cartesian product, Python UDF evaluation or a scan with no filter pushed down.

orders = spark.createDataFrame([], "order_id INT, customer_id INT, country STRING")
customers = spark.createDataFrame([], "customer_id INT, name STRING")
plan = orders.join(customers).filter("country = 'IN'")      # no join key: a cartesian product
plan.explain()   # the plan names the join type; a rule reports a cartesian product or nested-loop join

Checked on Spark 3.5.1: with empty inputs, the physical plan shows CartesianProduct, without reading any data.

It can't see sizes, so it can't judge broadcast choices or skew. Those need statistics or a real run.

FAQ

Questions people ask

What is an abstract syntax tree?

The structure of code after parsing: each call, condition, join and loop becomes a node in a tree. Linters walk the tree, which is far more reliable than searching the text.

Is there a SQL parser for Databricks SQL?

Yes. sqlglot parses the Databricks dialect into a syntax tree and is used by Databricks' own UCX and Lakebridge tools. SQLFluff also has a databricks dialect for style rules.

How can I check a PySpark plan without data?

Create empty DataFrames with the real schemas, run the transformation and call explain(). The physical plan shows join types, Python UDF evaluation and pushed filters in seconds.

Next

Where to go next

Next page: Pure functions in PySpark: possible, and how we detect them

↑ All of shift left · The repo on GitHub · Data project review