Home › Data engineering › Streaming › Real-time mode

Data engineering · free guide · one page

Databricks real time: real-time mode and Lakehouse//RT

The fast lane in the streaming design puts plant readings on a live board within seconds using Flink, DynamoDB and a WebSocket. Databricks now has two real-time pieces: real-time mode, which moves data in milliseconds (generally available), and Lakehouse//RT, a new engine that answers queries in milliseconds (beta). This page shows what each does, what it cannot do, what it costs, and where it fits.

In plain words

What real-time mode is

Normal Spark streaming collects records for a moment and processes them as a small batch. Real-time mode processes each record as it arrives, so the delay drops from seconds to milliseconds. It is the same Spark code with a different trigger.

What it isA trigger type for Spark Structured Streaming: records are processed as they arrive instead of in micro-batches. Databricks quotes latency under 100 ms, and as low as a few milliseconds.
StatusStructured Streaming jobs: announced generally available in March 2026. Lakeflow (Spark Declarative) pipelines: generally available on 29 October 2026.
RuntimeDatabricks Runtime 16.4 LTS or later.
ComputeJobs: classic compute with a fixed size (autoscaling off, Photon off, enough task slots for every task). Pipelines: serverless, no sizing, pay for the cores the query uses; the pipeline runs in continuous mode.

The picture

The 1-second lane, two ways

AWS-native (today)Kinesisevent log, 7 daysPULLManaged Flinkwindows, limitsPUSHDynamoDBlive store, 48 hPULLLambdareads the streamPUSHWebSocketto the live boardDatabricks onlyKinesisor Kafka, MSKPULLReal-time modefixed classic clusterPUSHLakebasePostgres, in UCPULLDatabricks Apppolls every secondPUSHBrowserserver-sent events Real-time mode cannot write Delta, so even the Databricks-only lane needs a non-Delta live store (Lakebase here).

Tags show who starts each transfer: PUSH, the sender; PULL, the receiver.

The new engine

Lakehouse//RT: fast queries, not fresher data

Real-time mode moves data quickly. Lakehouse//RT answers questions quickly. They solve different problems, and a 1-second design may use both.

What it isA new SQL warehouse type, powered by a new engine called Reyden, for read-only queries at very low latency and high concurrency on Delta and Iceberg tables in Unity Catalog. No separate serving copy.
ClaimsAs low as 10 ms on small datasets, under 100 ms on larger ones, under 100 ms at 12,000 queries a second on standard benchmarks. One independent test found it consistently faster than a normal serverless warehouse but not close to 100 ms; measure on your data.
StatusBeta since 30 June 2026, for selected read-only workloads; ask your Databricks account team. AI/BI dashboards can use it (beta, October 2026).
PriceNo list price published; 30% introductory discount through January 2027.
What it does not doIt does not make data arrive faster. A screen is as fresh as the table it reads; Lakehouse//RT only makes the query fast. It does not write (no INSERT, MERGE or ETL).

Where it fits in the streaming design

NeedWith Lakehouse//RTVerdict
Fast, many-user dashboards and apps on gold or silver (slicing, filters, drill-down)Lakehouse//RT instead of ClickHouse: same speed class, governed in Unity Catalog, no copy to keep in line with goldPursue once it leaves beta
Power BI DirectQuery on goldPoint it at a Lakehouse//RT warehouse for fast visuals under load; check connector support during the betaTry in the beta
Live board with data seconds oldNot on its own: tables are updated by pipelines in minutes (or about a minute if continuous). Seconds still need real-time mode into Lakebase, or the AWS-native laneDrop as a fast-lane replacement
Platform lane made faster (chained or continuous) plus Lakehouse//RTScreen age stays the pipeline's (6 to 22 min chained, 1 to 3 min continuous), but every query answers in milliseconds for hundreds of usersPursue when concurrency, not freshness, is the pain
The all-Databricks real-time picture: real-time mode in a serverless pipeline writes the latest values to Lakebase for the live board (seconds); the platform lane keeps silver and gold in Delta (minutes); Lakehouse//RT serves dashboards and apps on gold in milliseconds. Everything sits in Unity Catalog. The AWS-native fast lane is still cheaper and gives true push.

Sources and sinks

What it can read and write

SystemAs a sourceAs a sink
Kafka (and MSK, Event Hubs through its Kafka endpoint)YesYes
KinesisYes (enhanced fan-out recommended; polling only on newer runtimes, check)No
Delta tablesNoNo
Pub/Sub, PulsarNoNo
Custom writer (forEach), e.g. to Lakebase, Redis or DynamoDB-Yes (a native Lakebase sink is announced as future work)
forEachBatch-No

Limits

What to know before you choose it

LimitWhat it means here
Cannot write DeltaThe live store must be something else: Lakebase, DynamoDB, Kafka. Bronze still comes from the platform lane.
Fixed, always-on clusterNo autoscaling and no scale to zero; you size for the peak and pay all day.
Update output mode onlyEach output is an upsert of the latest value, which suits a live store.
No event-time timers, no transformWithStateInPandasSome stateful patterns (alerts after N minutes of silence) need another approach.
No PhotonCPU sizing as plain Spark.

Cost

Rough cost a month, side by side

AWS-native laneDatabricks-only lane
ProcessingManaged Flink, 2 + 1 units: about $250Real-time mode, driver + 1 to 2 workers (EC2 + jobs DBUs): about $430 to $650
Live storeDynamoDB, changes only: about $40 to $50Lakebase: about $100 to $300
Push to screensLambda + API Gateway WebSocket: about $130Databricks App: about $50 to $150
Simple limit alertsLambda: about $8Inside the same query
Total, production example (300 readings a second)about $530 a monthabout $700 to $1,100 a month
Governed in Unity CatalogNo (IAM on AWS)Yes, end to end
Push or pull into the screenPush (DynamoDB Streams then WebSocket)The app pulls from Lakebase every second, then pushes to the browser

Rough US list prices from 2025 and 2026, production example (5 plants, 300 readings a second). Databricks publishes no fixed price for this setup; measure it on your workload. More in the cost calculator.

Governance

A live store inside Unity Catalog

DynamoDB is not governed by Unity Catalog and is not a Lakehouse Federation source. If policy says every store must be in Unity Catalog, the live store becomes Lakebase.

Live storeIn Unity Catalog?ProsConsVerdict
DynamoDBNo; not a Lakehouse Federation source eitherPush via Streams; about $50; no servers to runOutside Unity CatalogPursue if a disposable 48-hour copy may sit outside Unity Catalog
Lakebase (Postgres run by Databricks)Yes: registered as a catalog with Unity Catalog permissionsSQL; same platform; can also be the metadata storeNo free push (poll, or LISTEN/NOTIFY, which loses messages while disconnected); always on, about $100 to $300Pursue if every store must be in Unity Catalog
RDS or Aurora Postgres with Lakehouse FederationOnly queries made from DatabricksFamiliarThe live app reads it directly, outside Unity CatalogDrop: governance on paper only
A Delta table as the live storeYesFully governedNot built for per-second single-row updates; real-time mode cannot write it anywayDrop
The live store holds a 48-hour copy that expires. The record of the same readings is governed in bronze, silver and gold. That is why keeping DynamoDB outside Unity Catalog is usually acceptable.

Two ways to run it

A Structured Streaming job or a Lakeflow pipeline

Both use the same real-time mode engine for the same Databricks-only lane. They differ in how you write the code and who runs the machines.

Structured Streaming jobLakeflow pipeline
How you write itOrdinary PySpark: readStream, your logic, writeStream with the real-time trigger, run as a Lakeflow JobDeclarative: a pipeline of flows and sinks, with real-time mode switched on for the pipeline
ComputeA classic cluster you size: fixed size, autoscaling off, Photon off, enough task slots for every taskServerless: Databricks picks and manages the machines
Who tunes itYou: machine type, number of workers, task slotsDatabricks
BillingEC2 plus jobs DBUs, all day (about $430 to $650 a month in the example)Serverless, paying for the cores the query uses, all day (continuous mode); no published list figure, measure it
StatusAnnounced generally available, March 2026Generally available, 29 October 2026
The same in bothEngine, sources (Kafka, Kinesis), no Delta sink, Lakebase live store, the app polling itSame

Choosing

Verdict

OptionProsConsVerdict
Keep the AWS-native lane (Flink, DynamoDB, WebSocket)Cheapest, true push, millisecondsTwo engines (Flink and Spark); the live store is outside Unity CatalogPursue (default)
Databricks-only lane, real-time mode in a Structured Streaming job (classic cluster)One platform, governed end to end, one engine; the proven path (announced GA)About $200 to $600 a month more; you size and run a fixed cluster; no Delta sink; the app polls LakebasePursue only if one governed platform is worth the extra cost
Databricks-only lane, real-time mode in a Lakeflow pipeline (serverless)Same engine and lane, GA since 29 Oct 2026, no cluster to size or tuneEach real-time query gets its own resources; writes go through a custom writer (no native Lakebase sink yet); still always on; the app polls LakebasePursue this one if you go Databricks-only (prefer it to the job version)

FAQ

Questions people ask

Is Databricks real-time mode generally available?

Yes. For Structured Streaming jobs it was announced generally available in March 2026 (Runtime 16.4 LTS and later), and for Lakeflow (Spark Declarative) pipelines on 29 October 2026.

What is Lakehouse//RT?

A new Databricks SQL warehouse type (engine: Reyden), in beta since June 2026, that answers read-only queries on Delta and Iceberg tables in milliseconds at high concurrency. It makes queries fast; it does not make data fresher.

Can real-time mode write to Delta tables?

No. Sinks are Kafka and custom forEach writers. For a live store, write to Lakebase or DynamoDB; keep bronze, silver and gold in the platform lane.

Does real-time mode work with Kinesis?

Yes, as a source, with enhanced fan-out recommended. Kinesis is not supported as a sink.

How much does a 1-second lane on Databricks cost?

Roughly $700 to $1,100 a month for a small plant stream: a fixed classic cluster, Lakebase and a Databricks App. The AWS-native lane (Flink, DynamoDB, WebSocket) is about $530. Rough list prices; measure.

Is DynamoDB governed by Unity Catalog?

No. If every store must be in Unity Catalog, use Lakebase as the live store and accept that the app polls it.

Sources

Checked against

Related

Where it connects

Streaming guide · Streaming simulator (pick the "Databricks only" binding) · A faster platform lane · Delta vs Iceberg