Home › Learn › Databricks compute and cost › Photon and caches
Databricks compute and cost · data engineering
Photon and every Databricks cache: result, Spark, disk, page and metadata caches
How Photon replaces Spark's JVM operators with a vectorized C++ engine, when it helps, and which cache each compute type supports: result cache, Spark cache, disk cache, OS page cache, Delta metadata cache.
Prices, defaults, limits and preview status change often, and dollar figures here are illustrations. Check the current Databricks documentation and price list before quoting a number.
Photon and caches
Photon: the faster engine
Normal Spark executes your query inside the Java JVM, one row or one generated code block at a time. Photon is a replacement execution engine written in C++. It processes data in columns, in batches, using the CPU's vector (SIMD) instructions. Same Spark, same APIs, same driver and plan, just a faster engine at the bottom.
| Photon helps a lot | Photon helps little or not at all |
|---|---|
| Large scans of Delta and Parquet Joins and aggregations MERGE, UPDATE, DELETE, big writes Wide tables with many columns SQL warehouses (it is always on) | Python UDFs and pandas UDFs RDD code and low-level Spark Very small queries that finish in a second or two Machine learning training Work limited by slow external systems, not CPU |
On classic clusters you tick a Photon box and the machine rate in DBUs goes up. It usually still saves money because jobs finish sooner. On SQL warehouses and serverless it is simply included.
Photon and caches
Every cache, and who supports it
Databricks has several different things people call "cache". They live in different places and solve different problems.
| Cache | Stores | Lives in | All-purpose / job cluster | Classic / Pro SQL warehouse | Serverless SQL warehouse | Serverless notebook / job |
|---|---|---|---|---|---|---|
| Result cache | Final query results | Warehouse memory, remote store for serverless | No | Yes, local | Yes, local and remote (up to 24 h) | No |
Spark cache (cache(), persist()) | DataFrame partitions | Executor RAM, can spill to disk | Yes, you manage it | Not something you control | Not something you control | Restricted |
| Disk cache (old name: Delta cache) | Copies of Parquet / Delta files | Worker local SSD | Yes. Automatic on storage-optimized types, opt-in on others. | Yes, automatic | Yes, managed | Yes, managed |
| OS page cache | Recently read file blocks | Spare RAM | Yes | Yes | Yes | Yes |
| Delta log / metadata cache | Table snapshot, file stats | Driver memory | Yes | Yes | Yes | Yes |
| Predictive IO (a read optimizer, not a cache) | n/a | n/a | No | Pro only | Yes | Not a documented feature |
Disk cache versus Spark cache, the part everyone mixes up. The disk cache keeps raw data files on local SSD and is invalidated automatically when files change. It costs no RAM and needs no code. The Spark cache keeps a specific DataFrame in memory after your transformations, uses your RAM, and you must unpersist it. For most workloads, rely on the disk cache and use Spark cache only for a DataFrame you reuse many times in one session.
FAQ
Questions people ask
Does Photon speed up everything?
No. It speeds up scans, joins, aggregations and big writes. It does nothing for Python UDFs, RDD code, very small queries or ML training.
Disk cache or Spark cache?
The disk cache keeps data file copies on local SSD, is automatic and costs no RAM. The Spark cache keeps a specific DataFrame in memory and you manage it. Rely on the disk cache by default.
Next
Where to go next
Related on this site: AQE, Photon, Predictive Optimization · Biggest cost and performance levers
All Databricks compute and cost pages · All data engineering · Databricks on this site