Home › Learn › Databricks interview prep › AQE, Photon, PO
Databricks interview prep · data engineering
AQE, Photon, Predictive Optimization and newer features
What AQE, Photon and Predictive Optimization do, when each helps or does not, how Photon works inside, and a map of newer Databricks features.
Defaults, limits and preview status change often. Check the current Databricks docs before quoting a number.
Four different layers: planning, re-planning, execution, maintenance.
AQE
Adaptive Query Execution
Spark normally builds one plan from guesses about data size. AQE re-plans between stages using the real sizes. It is in open-source Spark 3 and on by default on Databricks.
| What it does | Effect |
|---|---|
| Coalesces small shuffle partitions | Fixes a badly chosen shuffle.partitions |
| Switches join strategy at run time | A sort-merge join becomes a broadcast when a side turns out small |
| Splits skewed partitions | Handles skew in sort-merge joins |
| Dynamic partition and file pruning (related) | Uses the other side's values to skip data |
Limits: it decides only at shuffle boundaries, cannot fix bad file layout, and the plan before the run can differ from what ran. Read the final plan in the query profile.
Predictive Optimization
PO
A Databricks service that maintains Unity Catalog managed tables: OPTIMIZE, VACUUM and ANALYZE, run when its cost model says it pays. Related: automatic liquid clustering (CLUSTER BY AUTO) picks clustering keys from query history.
- Billed as serverless usage; watch it in system tables.
- Does not cover external tables.
- You give up control of timing.
- It does not fix a bad design such as a streaming trigger that makes tiny files.
Photon
When to use it and when not
Photon is a native C++ vectorized engine that replaces Spark's JVM execution for supported operators. Catalyst, scheduling and the APIs stay Spark, and results are the same with or without it.
| Use it for | Skip it for |
|---|---|
| SQL and DataFrame ETL: scans, filters, joins, aggregations | Python UDFs and RDD code: nothing to speed up |
| Large joins and group-bys | Tiny jobs dominated by startup |
| Delta scans, MERGE, UPDATE, DELETE | Jobs bound by an outside API, network or driver work |
| BI and dashboards on SQL warehouses | ML training and non-SQL libraries |
| CPU-bound jobs | Queries that fall back to Spark for most operators |
It costs more DBUs per hour, so judge cost per job, not per hour. In the query profile, look for operators labelled Photon and for fallback boundaries.
How it works
Photon inside
| Technique | Meaning |
|---|---|
| Vectorized execution | Processes batches of column values, not one row at a time; cache friendly and uses SIMD instructions. |
| Columnar in-memory format | Data stays columnar from the Parquet read through the operators. |
| Native code, off-heap memory | No JVM garbage collection pauses or object overhead; memory still accounted through Spark's manager. |
| Vectorized interpretation, not code generation | Pre-compiled kernels run over vectors: easier to debug, better per-operator metrics, no compile time per query, easier to adapt. |
| Runtime adaptivity | Chooses a code path per batch (nulls or not, all rows active or not, ASCII strings). |
| Fast hash tables and I/O | Vectorized hashing and probing with prefetching; native Parquet decoding and writing. |
| Where it plugs in | A rule swaps supported operators bottom-up; at an unsupported operator the plan converts back to Spark. Each switch costs time. |
Newer things
Other recent features to know
Names and status change quickly. Check the release notes before quoting any of these as generally available.
| Area | Features |
|---|---|
| Storage and Delta | Liquid clustering and automatic liquid clustering, deletion vectors, row-level concurrency, column mapping, type widening, UniForm, managed Iceberg, change data feed, collations, the Variant type |
| Compute and engine | Serverless compute, Photon, materialized views with incremental refresh, Spark Connect, Lakehouse Federation, query result and disk caching |
| Ingestion and pipelines | Auto Loader, Lakeflow Declarative Pipelines, Lakeflow Connect, AUTO CDC, Lakeflow Jobs triggers, real-time mode, transformWithState |
| Governance | Unity Catalog, row filters and masks, ABAC, Metric Views, system tables, volumes, Delta Sharing, data quality monitoring, budget policies |
| AI and apps | Genie, AI functions in SQL, Vector Search, Model Serving, agent tooling, Databricks Apps, Lakebase |
| Developer experience | Asset Bundles, Databricks Connect, Databricks Assistant |
Already on this site: declarative pipelines, open source and Lakeflow, metric views, Delta vs Iceberg, who maintains your tables, real-time mode.
Practice
Questions to rehearse
- What can AQE fix, and what can it not?
- What does Predictive Optimization run, who pays, and where do you see it?
- What does Photon replace and what stays Spark?
- Why vectorized interpretation instead of code generation?
- A job got dearer after enabling Photon. Why might that be?
- How do you decide on Photon for 200 nightly jobs?
FAQ
Questions people ask
Does Photon change query results?
No. It keeps Spark semantics, so results are the same with or without it.
Does AQE replace statistics?
No. Good statistics still give better first plans and join order.
Next
Where to go next
All Databricks interview topics · All data engineering · Databricks on this site