Home › Learn › How reads and writes work › 7 · Photon

Delta Lake and Spark · from zero to advanced

Photon: how it speeds up queries and when it does not

What the Photon engine replaces, why a vectorised C++ engine beats row-at-a-time JVM execution, which operators it covers, where it cannot help, and how to see it in a plan. Written from the Databricks docs; Photon cannot run locally so it was not measured.

Read this first. Photon only exists on Databricks, so nothing on this page was measured on the laptop. It is written from the Databricks documentation, and the one local observation (the open-source plan) is marked. Where the reason is general knowledge rather than from the docs, it says so.

The problem

Spark on the JVM spends a lot of time not computing

Open-source Spark runs its operators on the JVM. For a long time the main cost was doing the arithmetic on one row at a time, with objects allocated for rows, garbage to collect, and a function call per row for every step. As disks and networks got faster, the CPU became the bottleneck, and that overhead stood out.

One row at a time through each stepA batch of values through each stepscanfilterprojectaggregaterow 1row 2row 3row 4row 5row 6each row pays the call overhead of every stepscanfilterprojectaggregatea whole column of values, processed in a tight loopthe call overhead is paid once per batch

The idea behind vectorised engines: pay the overhead once per batch, not once per row.

How it solves it

A native, vectorised engine in place of the JVM execution layer

Your querySQL / DataFrameSpark's planner (Catalyst)still plans the query, in the JVMPhotonnative C++ engine runsthe supported operatorsPlain Sparkruns anything Photoncannot (falls back)Per the Databricks docs: Photon replaces the JVM execution layer for supported operations. Unsupported operations fall back to Spark.

Photon replaces the execution, not the planning.

What changes (Databricks docs)Effect
Native C++ vectorised engine replaces the JVM execution layerNo per-row JVM overhead for supported operators
Replaces sort-merge joins with hash joinsCheaper joins (see the joins page)
Columnar shuffleData stays column-shaped when it moves between executors
Scan: filter pushdown, dictionary pruning, row-group skippingLess data decoded at the file reader
Predictive I/O requires PhotonFaster selective reads on supported tables
Dynamic file pruning in MERGE, UPDATE, DELETE requires PhotonFewer files rewritten (see the write page)
  • Why native code is faster (general knowledge, not in the docs I read): no garbage collector, no object per row, and tight loops over arrays that CPUs handle well. The docs say it is vectorised; I did not read a figure for any speed-up.
  • Local observation, open-source Spark: the real plans show FileScan parquet … Batched: true followed by ColumnarToRow. Open-source Spark reads Parquet in column batches but then converts to rows for most later steps. Photon's pitch is to avoid that conversion for supported operators; I could not verify that locally.

What it covers

Operators, and what it cannot do

Covered (docs)Not covered (docs)
Scans (Parquet, Delta, CSV, JSON), filter, project, hash aggregate, hash join, hash shuffle, nested-loop join, union, sort, top-k, limit, window, Delta and Parquet writeUDFs, RDD APIs, Dataset APIs
Null-aware anti join, spatial join, expand, scalar subqueryStateful streaming
Queries shorter than about 2 seconds see no meaningful benefit
  • Where it is on: by default for serverless, SQL warehouses and serverless pipelines. On classic compute, jobs and classic pipelines it is on by default with a Use Photon Acceleration switch; the APIs use runtime_engine = PHOTON.

Seeing it

How to tell Photon ran

  • Spark UI: Photon operators show in orange in the DAG.
  • Query profile: Photon operators show in purple, and the profile gives the percentage of task time spent in Photon.
  • If a step is not Photon, it ran in regular Spark. A Python UDF in the middle is the usual reason.

A query that is mostly Photon operators with one UDF in the middle will fall back for that step and pay the conversion in and out.

When it helps

A short guide

WorkloadExpectVerdict
Big scans, joins, aggregations, MERGE on DeltaBiggest gains (per the docs' operator list)Pursue
Short queries under about 2 secondsNo meaningful benefitDrop
Heavy use of Python UDFs or RDD codeFalls back to SparkDrop
Anything waiting on slow storage rather than CPULittle to gain (general reasoning, not from the docs)Drop

See also the cost side in the Photon and caches compute page.

FAQ

Questions people ask

What does Photon replace?

The JVM execution layer for supported operators. Spark's Catalyst optimizer still plans the query; Photon then runs the supported operators in a native C++ vectorised engine, and unsupported operations fall back to regular Spark.

When does Photon not help?

Per the Databricks docs it does not run UDFs, RDD or Dataset APIs, or stateful streaming, and queries under about two seconds see no meaningful benefit.

How can I see whether a query used Photon?

In the Spark UI Photon operators appear in orange, and in the query profile they appear in purple with the share of task time spent in Photon.

← 6 · Python and JVM8 · Plan and UI →