Home › Data engineering › Streaming › Real-time mode
Data engineering · free guide · one page
Databricks real time: real-time mode and Lakehouse//RT
The fast lane in the streaming design puts plant readings on a live board within seconds using Flink, DynamoDB and a WebSocket. Databricks now has two real-time pieces: real-time mode, which moves data in milliseconds (generally available), and Lakehouse//RT, a new engine that answers queries in milliseconds (beta). This page shows what each does, what it cannot do, what it costs, and where it fits.
In plain words
What real-time mode is
Normal Spark streaming collects records for a moment and processes them as a small batch. Real-time mode processes each record as it arrives, so the delay drops from seconds to milliseconds. It is the same Spark code with a different trigger.
| What it is | A trigger type for Spark Structured Streaming: records are processed as they arrive instead of in micro-batches. Databricks quotes latency under 100 ms, and as low as a few milliseconds. |
| Status | Structured Streaming jobs: announced generally available in March 2026. Lakeflow (Spark Declarative) pipelines: generally available on 29 October 2026. |
| Runtime | Databricks Runtime 16.4 LTS or later. |
| Compute | Jobs: classic compute with a fixed size (autoscaling off, Photon off, enough task slots for every task). Pipelines: serverless, no sizing, pay for the cores the query uses; the pipeline runs in continuous mode. |
The picture
The 1-second lane, two ways
Tags show who starts each transfer: PUSH, the sender; PULL, the receiver.
The new engine
Lakehouse//RT: fast queries, not fresher data
Real-time mode moves data quickly. Lakehouse//RT answers questions quickly. They solve different problems, and a 1-second design may use both.
| What it is | A new SQL warehouse type, powered by a new engine called Reyden, for read-only queries at very low latency and high concurrency on Delta and Iceberg tables in Unity Catalog. No separate serving copy. |
| Claims | As low as 10 ms on small datasets, under 100 ms on larger ones, under 100 ms at 12,000 queries a second on standard benchmarks. One independent test found it consistently faster than a normal serverless warehouse but not close to 100 ms; measure on your data. |
| Status | Beta since 30 June 2026, for selected read-only workloads; ask your Databricks account team. AI/BI dashboards can use it (beta, October 2026). |
| Price | No list price published; 30% introductory discount through January 2027. |
| What it does not do | It does not make data arrive faster. A screen is as fresh as the table it reads; Lakehouse//RT only makes the query fast. It does not write (no INSERT, MERGE or ETL). |
Where it fits in the streaming design
| Need | With Lakehouse//RT | Verdict |
|---|---|---|
| Fast, many-user dashboards and apps on gold or silver (slicing, filters, drill-down) | Lakehouse//RT instead of ClickHouse: same speed class, governed in Unity Catalog, no copy to keep in line with gold | Pursue once it leaves beta |
| Power BI DirectQuery on gold | Point it at a Lakehouse//RT warehouse for fast visuals under load; check connector support during the beta | Try in the beta |
| Live board with data seconds old | Not on its own: tables are updated by pipelines in minutes (or about a minute if continuous). Seconds still need real-time mode into Lakebase, or the AWS-native lane | Drop as a fast-lane replacement |
| Platform lane made faster (chained or continuous) plus Lakehouse//RT | Screen age stays the pipeline's (6 to 22 min chained, 1 to 3 min continuous), but every query answers in milliseconds for hundreds of users | Pursue when concurrency, not freshness, is the pain |
Sources and sinks
What it can read and write
| System | As a source | As a sink |
|---|---|---|
| Kafka (and MSK, Event Hubs through its Kafka endpoint) | Yes | Yes |
| Kinesis | Yes (enhanced fan-out recommended; polling only on newer runtimes, check) | No |
| Delta tables | No | No |
| Pub/Sub, Pulsar | No | No |
| Custom writer (forEach), e.g. to Lakebase, Redis or DynamoDB | - | Yes (a native Lakebase sink is announced as future work) |
| forEachBatch | - | No |
Limits
What to know before you choose it
| Limit | What it means here |
|---|---|
| Cannot write Delta | The live store must be something else: Lakebase, DynamoDB, Kafka. Bronze still comes from the platform lane. |
| Fixed, always-on cluster | No autoscaling and no scale to zero; you size for the peak and pay all day. |
| Update output mode only | Each output is an upsert of the latest value, which suits a live store. |
| No event-time timers, no transformWithStateInPandas | Some stateful patterns (alerts after N minutes of silence) need another approach. |
| No Photon | CPU sizing as plain Spark. |
Cost
Rough cost a month, side by side
| AWS-native lane | Databricks-only lane | |
|---|---|---|
| Processing | Managed Flink, 2 + 1 units: about $250 | Real-time mode, driver + 1 to 2 workers (EC2 + jobs DBUs): about $430 to $650 |
| Live store | DynamoDB, changes only: about $40 to $50 | Lakebase: about $100 to $300 |
| Push to screens | Lambda + API Gateway WebSocket: about $130 | Databricks App: about $50 to $150 |
| Simple limit alerts | Lambda: about $8 | Inside the same query |
| Total, production example (300 readings a second) | about $530 a month | about $700 to $1,100 a month |
| Governed in Unity Catalog | No (IAM on AWS) | Yes, end to end |
| Push or pull into the screen | Push (DynamoDB Streams then WebSocket) | The app pulls from Lakebase every second, then pushes to the browser |
Rough US list prices from 2025 and 2026, production example (5 plants, 300 readings a second). Databricks publishes no fixed price for this setup; measure it on your workload. More in the cost calculator.
Governance
A live store inside Unity Catalog
DynamoDB is not governed by Unity Catalog and is not a Lakehouse Federation source. If policy says every store must be in Unity Catalog, the live store becomes Lakebase.
| Live store | In Unity Catalog? | Pros | Cons | Verdict |
|---|---|---|---|---|
| DynamoDB | No; not a Lakehouse Federation source either | Push via Streams; about $50; no servers to run | Outside Unity Catalog | Pursue if a disposable 48-hour copy may sit outside Unity Catalog |
| Lakebase (Postgres run by Databricks) | Yes: registered as a catalog with Unity Catalog permissions | SQL; same platform; can also be the metadata store | No free push (poll, or LISTEN/NOTIFY, which loses messages while disconnected); always on, about $100 to $300 | Pursue if every store must be in Unity Catalog |
| RDS or Aurora Postgres with Lakehouse Federation | Only queries made from Databricks | Familiar | The live app reads it directly, outside Unity Catalog | Drop: governance on paper only |
| A Delta table as the live store | Yes | Fully governed | Not built for per-second single-row updates; real-time mode cannot write it anyway | Drop |
Two ways to run it
A Structured Streaming job or a Lakeflow pipeline
Both use the same real-time mode engine for the same Databricks-only lane. They differ in how you write the code and who runs the machines.
| Structured Streaming job | Lakeflow pipeline | |
|---|---|---|
| How you write it | Ordinary PySpark: readStream, your logic, writeStream with the real-time trigger, run as a Lakeflow Job | Declarative: a pipeline of flows and sinks, with real-time mode switched on for the pipeline |
| Compute | A classic cluster you size: fixed size, autoscaling off, Photon off, enough task slots for every task | Serverless: Databricks picks and manages the machines |
| Who tunes it | You: machine type, number of workers, task slots | Databricks |
| Billing | EC2 plus jobs DBUs, all day (about $430 to $650 a month in the example) | Serverless, paying for the cores the query uses, all day (continuous mode); no published list figure, measure it |
| Status | Announced generally available, March 2026 | Generally available, 29 October 2026 |
| The same in both | Engine, sources (Kafka, Kinesis), no Delta sink, Lakebase live store, the app polling it | Same |
Choosing
Verdict
| Option | Pros | Cons | Verdict |
|---|---|---|---|
| Keep the AWS-native lane (Flink, DynamoDB, WebSocket) | Cheapest, true push, milliseconds | Two engines (Flink and Spark); the live store is outside Unity Catalog | Pursue (default) |
| Databricks-only lane, real-time mode in a Structured Streaming job (classic cluster) | One platform, governed end to end, one engine; the proven path (announced GA) | About $200 to $600 a month more; you size and run a fixed cluster; no Delta sink; the app polls Lakebase | Pursue only if one governed platform is worth the extra cost |
| Databricks-only lane, real-time mode in a Lakeflow pipeline (serverless) | Same engine and lane, GA since 29 Oct 2026, no cluster to size or tune | Each real-time query gets its own resources; writes go through a custom writer (no native Lakebase sink yet); still always on; the app polls Lakebase | Pursue this one if you go Databricks-only (prefer it to the job version) |
FAQ
Questions people ask
Is Databricks real-time mode generally available?
Yes. For Structured Streaming jobs it was announced generally available in March 2026 (Runtime 16.4 LTS and later), and for Lakeflow (Spark Declarative) pipelines on 29 October 2026.
What is Lakehouse//RT?
A new Databricks SQL warehouse type (engine: Reyden), in beta since June 2026, that answers read-only queries on Delta and Iceberg tables in milliseconds at high concurrency. It makes queries fast; it does not make data fresher.
Can real-time mode write to Delta tables?
No. Sinks are Kafka and custom forEach writers. For a live store, write to Lakebase or DynamoDB; keep bronze, silver and gold in the platform lane.
Does real-time mode work with Kinesis?
Yes, as a source, with enhanced fan-out recommended. Kinesis is not supported as a sink.
How much does a 1-second lane on Databricks cost?
Roughly $700 to $1,100 a month for a small plant stream: a fixed classic cluster, Lakebase and a Databricks App. The AWS-native lane (Flink, DynamoDB, WebSocket) is about $530. Rough list prices; measure.
Is DynamoDB governed by Unity Catalog?
No. If every store must be in Unity Catalog, use Lakebase as the live store and accept that the app polls it.
Sources
Checked against
- Real-time mode in Structured Streaming
- Real-time mode reference (sources, sinks, limits)
- General availability announcement (Structured Streaming)
- General availability in Spark Declarative Pipelines (29 Oct 2026)
- Introducing Lakehouse//RT
- Lakehouse//RT press release
- An independent Lakehouse//RT test (Eraneos)
Related
Where it connects
Streaming guide · Streaming simulator (pick the "Databricks only" binding) · A faster platform lane · Delta vs Iceberg