Home › Learn › How reads and writes work › 7 · Photon
Delta Lake and Spark · from zero to advanced
Photon: how it speeds up queries and when it does not
What the Photon engine replaces, why a vectorised C++ engine beats row-at-a-time JVM execution, which operators it covers, where it cannot help, and how to see it in a plan. Written from the Databricks docs; Photon cannot run locally so it was not measured.
The problem
Spark on the JVM spends a lot of time not computing
Open-source Spark runs its operators on the JVM. For a long time the main cost was doing the arithmetic on one row at a time, with objects allocated for rows, garbage to collect, and a function call per row for every step. As disks and networks got faster, the CPU became the bottleneck, and that overhead stood out.
The idea behind vectorised engines: pay the overhead once per batch, not once per row.
How it solves it
A native, vectorised engine in place of the JVM execution layer
Photon replaces the execution, not the planning.
| What changes (Databricks docs) | Effect |
|---|---|
| Native C++ vectorised engine replaces the JVM execution layer | No per-row JVM overhead for supported operators |
| Replaces sort-merge joins with hash joins | Cheaper joins (see the joins page) |
| Columnar shuffle | Data stays column-shaped when it moves between executors |
| Scan: filter pushdown, dictionary pruning, row-group skipping | Less data decoded at the file reader |
| Predictive I/O requires Photon | Faster selective reads on supported tables |
| Dynamic file pruning in MERGE, UPDATE, DELETE requires Photon | Fewer files rewritten (see the write page) |
- Why native code is faster (general knowledge, not in the docs I read): no garbage collector, no object per row, and tight loops over arrays that CPUs handle well. The docs say it is vectorised; I did not read a figure for any speed-up.
- Local observation, open-source Spark: the real plans show
FileScan parquet … Batched: truefollowed byColumnarToRow. Open-source Spark reads Parquet in column batches but then converts to rows for most later steps. Photon's pitch is to avoid that conversion for supported operators; I could not verify that locally.
What it covers
Operators, and what it cannot do
| Covered (docs) | Not covered (docs) |
|---|---|
| Scans (Parquet, Delta, CSV, JSON), filter, project, hash aggregate, hash join, hash shuffle, nested-loop join, union, sort, top-k, limit, window, Delta and Parquet write | UDFs, RDD APIs, Dataset APIs |
| Null-aware anti join, spatial join, expand, scalar subquery | Stateful streaming |
| Queries shorter than about 2 seconds see no meaningful benefit |
- Where it is on: by default for serverless, SQL warehouses and serverless pipelines. On classic compute, jobs and classic pipelines it is on by default with a Use Photon Acceleration switch; the APIs use
runtime_engine = PHOTON.
Seeing it
How to tell Photon ran
- Spark UI: Photon operators show in orange in the DAG.
- Query profile: Photon operators show in purple, and the profile gives the percentage of task time spent in Photon.
- If a step is not Photon, it ran in regular Spark. A Python UDF in the middle is the usual reason.
A query that is mostly Photon operators with one UDF in the middle will fall back for that step and pay the conversion in and out.
When it helps
A short guide
| Workload | Expect | Verdict |
|---|---|---|
| Big scans, joins, aggregations, MERGE on Delta | Biggest gains (per the docs' operator list) | Pursue |
| Short queries under about 2 seconds | No meaningful benefit | Drop |
| Heavy use of Python UDFs or RDD code | Falls back to Spark | Drop |
| Anything waiting on slow storage rather than CPU | Little to gain (general reasoning, not from the docs) | Drop |
See also the cost side in the Photon and caches compute page.
FAQ
Questions people ask
What does Photon replace?
The JVM execution layer for supported operators. Spark's Catalyst optimizer still plans the query; Photon then runs the supported operators in a native C++ vectorised engine, and unsupported operations fall back to regular Spark.
When does Photon not help?
Per the Databricks docs it does not run UDFs, RDD or Dataset APIs, or stateful streaming, and queries under about two seconds see no meaningful benefit.
How can I see whether a query used Photon?
In the Spark UI Photon operators appear in orange, and in the query profile they appear in purple with the share of task time spent in Photon.