Home › Learn › Databricks compute and cost › Photon and caches

Databricks compute and cost · data engineering

Photon and every Databricks cache: result, Spark, disk, page and metadata caches

How Photon replaces Spark's JVM operators with a vectorized C++ engine, when it helps, and which cache each compute type supports: result cache, Spark cache, disk cache, OS page cache, Delta metadata cache.

Cluster anatomyWarehouses and serverlessChoosing computeCompute settingsCost by workloadLogs and visibilityPhoton and cachesHow billing worksDividing the billSystem tablesSharing and regionsBackup architectureCost-saving hacks

Prices, defaults, limits and preview status change often, and dollar figures here are illustrations. Check the current Databricks documentation and price list before quoting a number.

Photon and caches

Photon: the faster engine

Normal Spark executes your query inside the Java JVM, one row or one generated code block at a time. Photon is a replacement execution engine written in C++. It processes data in columns, in batches, using the CPU's vector (SIMD) instructions. Same Spark, same APIs, same driver and plan, just a faster engine at the bottom.

Your query plan (the same either way) Scan Delta Filter Join Aggregate Python UDF Write Without Photon: every step in the Spark JVM JVM JVM JVM JVM JVM + Python JVM With Photon: supported steps run in C++, the rest stay in the JVM Photon Photon Photon Photon JVM + Python Photon Where the plan switches between Photon and the JVM, data is converted, which costs a little. A plan with many unsupported steps in the middle gains less. Photon works on off-heap memory, so it needs RAM outside the Java heap. Databricks handles the split for you.
Photon does not change what your query means. It only swaps the engine for each operator it supports. Anything it does not support, such as Python UDFs, runs on normal Spark.
Photon helps a lotPhoton helps little or not at all
Large scans of Delta and Parquet
Joins and aggregations
MERGE, UPDATE, DELETE, big writes
Wide tables with many columns
SQL warehouses (it is always on)
Python UDFs and pandas UDFs
RDD code and low-level Spark
Very small queries that finish in a second or two
Machine learning training
Work limited by slow external systems, not CPU

On classic clusters you tick a Photon box and the machine rate in DBUs goes up. It usually still saves money because jobs finish sooner. On SQL warehouses and serverless it is simply included.

Photon and caches

Every cache, and who supports it

Databricks has several different things people call "cache". They live in different places and solve different problems.

Your queryarrives 1. Result cachewhole answer savedSQL warehouseshit = done instantly 2. Spark cacheDataFrame in RAMclassic clusters, you askhit = skip the scan 3. Disk cachefile copies on SSDautomatic, most computehit = skip cloud read 4. Cloudstorage missmissmiss The OS page cache sits quietly under all of these and needs no setup.
Checks go from cheapest and most complete (whole answer) to most expensive (reading from cloud). The further left a hit happens, the more work is skipped.
CacheStoresLives inAll-purpose / job clusterClassic / Pro SQL warehouseServerless SQL warehouseServerless notebook / job
Result cacheFinal query resultsWarehouse memory, remote store for serverlessNoYes, localYes, local and remote (up to 24 h)No
Spark cache (cache(), persist())DataFrame partitionsExecutor RAM, can spill to diskYes, you manage itNot something you controlNot something you controlRestricted
Disk cache (old name: Delta cache)Copies of Parquet / Delta filesWorker local SSDYes. Automatic on storage-optimized types, opt-in on others.Yes, automaticYes, managedYes, managed
OS page cacheRecently read file blocksSpare RAMYesYesYesYes
Delta log / metadata cacheTable snapshot, file statsDriver memoryYesYesYesYes
Predictive IO (a read optimizer, not a cache)n/an/aNoPro onlyYesNot a documented feature

Disk cache versus Spark cache, the part everyone mixes up. The disk cache keeps raw data files on local SSD and is invalidated automatically when files change. It costs no RAM and needs no code. The Spark cache keeps a specific DataFrame in memory after your transformations, uses your RAM, and you must unpersist it. For most workloads, rely on the disk cache and use Spark cache only for a DataFrame you reuse many times in one session.

FAQ

Questions people ask

Does Photon speed up everything?

No. It speeds up scans, joins, aggregations and big writes. It does nothing for Python UDFs, RDD code, very small queries or ML training.

Disk cache or Spark cache?

The disk cache keeps data file copies on local SSD, is automatic and costs no RAM. The Spark cache keeps a specific DataFrame in memory and you manage it. Rely on the disk cache by default.

Next

Where to go next

Related on this site: AQE, Photon, Predictive Optimization · Biggest cost and performance levers

All Databricks compute and cost pages · All data engineering · Databricks on this site