Home › Learn › Databricks interview prep › AQE, Photon, PO

Databricks interview prep · data engineering

AQE, Photon, Predictive Optimization and newer features

What AQE, Photon and Predictive Optimization do, when each helps or does not, how Photon works inside, and a map of newer Databricks features.

Maintenance commandsRow-level securityCompute choiceJoinsSkew and shuffleAQE, Photon, POCost and performance leversDeclarative and semanticAsset BundlesLakeflow ConnectUniFormVector Searchdbt and ODCSEgress and ingressMaintenance in picturesClustering and compactionMaintenance strategySlow writesBackup and recoveryTaggingQuestion bank

Defaults, limits and preview status change often. Check the current Databricks docs before quoting a number.

Catalyst plans the queryAQE re-plans at run timePhoton runs supportedoperators nativelyPredictive Optimizationkeeps tables healthy

Four different layers: planning, re-planning, execution, maintenance.

AQE

Adaptive Query Execution

Spark normally builds one plan from guesses about data size. AQE re-plans between stages using the real sizes. It is in open-source Spark 3 and on by default on Databricks.

What it doesEffect
Coalesces small shuffle partitionsFixes a badly chosen shuffle.partitions
Switches join strategy at run timeA sort-merge join becomes a broadcast when a side turns out small
Splits skewed partitionsHandles skew in sort-merge joins
Dynamic partition and file pruning (related)Uses the other side's values to skip data

Limits: it decides only at shuffle boundaries, cannot fix bad file layout, and the plan before the run can differ from what ran. Read the final plan in the query profile.

Predictive Optimization

PO

A Databricks service that maintains Unity Catalog managed tables: OPTIMIZE, VACUUM and ANALYZE, run when its cost model says it pays. Related: automatic liquid clustering (CLUSTER BY AUTO) picks clustering keys from query history.

Photon

When to use it and when not

Photon is a native C++ vectorized engine that replaces Spark's JVM execution for supported operators. Catalyst, scheduling and the APIs stay Spark, and results are the same with or without it.

Use it forSkip it for
SQL and DataFrame ETL: scans, filters, joins, aggregationsPython UDFs and RDD code: nothing to speed up
Large joins and group-bysTiny jobs dominated by startup
Delta scans, MERGE, UPDATE, DELETEJobs bound by an outside API, network or driver work
BI and dashboards on SQL warehousesML training and non-SQL libraries
CPU-bound jobsQueries that fall back to Spark for most operators

It costs more DBUs per hour, so judge cost per job, not per hour. In the query profile, look for operators labelled Photon and for fallback boundaries.

How it works

Photon inside

TechniqueMeaning
Vectorized executionProcesses batches of column values, not one row at a time; cache friendly and uses SIMD instructions.
Columnar in-memory formatData stays columnar from the Parquet read through the operators.
Native code, off-heap memoryNo JVM garbage collection pauses or object overhead; memory still accounted through Spark's manager.
Vectorized interpretation, not code generationPre-compiled kernels run over vectors: easier to debug, better per-operator metrics, no compile time per query, easier to adapt.
Runtime adaptivityChooses a code path per batch (nulls or not, all rows active or not, ASCII strings).
Fast hash tables and I/OVectorized hashing and probing with prefetching; native Parquet decoding and writing.
Where it plugs inA rule swaps supported operators bottom-up; at an unsupported operator the plan converts back to Spark. Each switch costs time.

Newer things

Other recent features to know

Names and status change quickly. Check the release notes before quoting any of these as generally available.

AreaFeatures
Storage and DeltaLiquid clustering and automatic liquid clustering, deletion vectors, row-level concurrency, column mapping, type widening, UniForm, managed Iceberg, change data feed, collations, the Variant type
Compute and engineServerless compute, Photon, materialized views with incremental refresh, Spark Connect, Lakehouse Federation, query result and disk caching
Ingestion and pipelinesAuto Loader, Lakeflow Declarative Pipelines, Lakeflow Connect, AUTO CDC, Lakeflow Jobs triggers, real-time mode, transformWithState
GovernanceUnity Catalog, row filters and masks, ABAC, Metric Views, system tables, volumes, Delta Sharing, data quality monitoring, budget policies
AI and appsGenie, AI functions in SQL, Vector Search, Model Serving, agent tooling, Databricks Apps, Lakebase
Developer experienceAsset Bundles, Databricks Connect, Databricks Assistant

Already on this site: declarative pipelines, open source and Lakeflow, metric views, Delta vs Iceberg, who maintains your tables, real-time mode.

Practice

Questions to rehearse

  1. What can AQE fix, and what can it not?
  2. What does Predictive Optimization run, who pays, and where do you see it?
  3. What does Photon replace and what stays Spark?
  4. Why vectorized interpretation instead of code generation?
  5. A job got dearer after enabling Photon. Why might that be?
  6. How do you decide on Photon for 200 nightly jobs?
In an interview, say: AQE re-plans with real sizes, Photon executes faster, Predictive Optimization maintains tables. Each has a limit, and I check each with the query profile or system tables, not by guessing.

FAQ

Questions people ask

Does Photon change query results?

No. It keeps Spark semantics, so results are the same with or without it.

Does AQE replace statistics?

No. Good statistics still give better first plans and join order.

Next

Where to go next

All Databricks interview topics · All data engineering · Databricks on this site