Home › Data engineering › Streaming › All options

Data engineering · one table

Streaming options on AWS at a glance: fast lane and platform lane

Every way we looked at to get streaming data to a screen, in one table: 23 options, 4 recommended. Each row gives the whole path, how fresh the screen is, whether data is pushed or pulled to it, whether it sits in Unity Catalog, rough cost, the verdict and why, and a link to the details.

The short answer: fast lane F1 (AWS-native, 1 to 3 s, ~$530) plus platform lane P2 (chained, 6 to 22 min, ~$770), with P9 for irregular files and P11 for database changes. Switch the fast lane to F6 only if one governed platform is worth the extra cost; add P5 once Lakehouse//RT leaves beta and many users query gold.
#OptionPath, hop by hopDelay to the screenScreen gets data byIn Unity CatalogRough cost a monthVerdictWhyDetails
Fast lane: seconds, for acting now
F1AWS-native laneKinesis, Managed Flink (windows, limits), DynamoDB, DynamoDB Streams read by Lambda, API Gateway WebSocket, React live board1 to 3 sPushNo (IAM)~$530Pursue (default)Cheapest true push in milliseconds to seconds; serverless parts; the live store is a 48-hour copy, the record stays governed in gold.Simulator, AWS + Databricks
F2F1 with AppSync EventsAs F1, but the function publishes to an AppSync Events channel instead of tracking WebSocket connections1 to 3 sPushNo~$500TrySame lane with less code: AWS handles the fan-out to screens, no connections table.Streaming guide, push and alerts
F3Alerts only, no FlinkKinesis, Lambda limit checks, SNS alerts and a small live storeunder 1 sPushNo~$180Pursue if only limit alerts are neededNearly free, but no time windows, no drift detection, no state.Streaming guide, live processor
F4Flink into ClickHouseKinesis, Flink (or ClickPipes), ClickHouse; the gateway polls every second and pushes differences2 to 5 sPull, then pushNo~$750 to $3,500Pursue only if live screens need slicing beyond 48 hoursFast ad-hoc filters for many users, but a new vendor, no push, and a third copy of the numbers.Streaming guide, to the UI
F5Flink into LakebaseKinesis, Flink, Lakebase (Postgres in Databricks); the app polls Lakebase2 to 5 sPull, then pushYes (store)~$450 to $650Pursue if the live store must be in Unity CatalogKeeps the cheap Flink lane but moves the live store under Unity Catalog; loses free push from DynamoDB Streams.Real time page, live store
F6Databricks real-time mode, serverless pipelineKinesis or Kafka, Lakeflow pipeline in real-time mode (GA 29 Oct 2026), custom writer into Lakebase, Databricks App, browser1 to 3 sPull, then pushYes~$700 to $1,100Pursue if one governed platform is worth the extra costOne engine and one platform, governed end to end, no cluster to size; costs more, cannot write Delta, the app polls.Real time page
F7Databricks real-time mode, classic jobAs F6, but a Structured Streaming job on a fixed classic cluster1 to 3 sPull, then pushYes~$700 to $1,100Drop (prefer F6)Same result as F6, but you size and run a fixed always-on cluster.Real time page, two ways to run it
F8Lakehouse//RT as the fast lanePipelines into Delta, Lakehouse//RT warehouse, dashboardminutes (table freshness)PullYesbeta pricingDrop as a fast-lane replacementIt makes queries fast, not data fresh: a screen is only as fresh as the table it reads.Real time page, Lakehouse//RT
F9Power BI streaming or push datasetsProcessor pushes rows straight into Power BIsecondsPushNo-DropBeing retired (no new ones after 31 Oct 2027), rate-limited, and a second copy of the logic.Simulator, push or pull
F10Push with no live storeProcessor pushes straight to screensabout 1 sPushNo~$100DropA screen that opens or reconnects has nothing to show until the next change; no last-hour chart.Streaming guide, to the UI
F11Zerobus as the only entryDevices straight into a bronze Delta tableseconds to bronze-Yes-Drop for sensorsOne reader only: no shared event log for a live lane and the lake, no replay from the log.Streaming guide, front door
F12Timestream LiveAnalytics, IoT EventsAWS time-series store and detector service--No-DropTimestream LiveAnalytics is closed to new customers; IoT Events reached end of support on 20 May 2026.Streaming guide, notes
Platform lane: minutes, the number of record, to Power BI
P1Today: hourly gold, Pro refresh15-min scheduled Lakeflow pipeline (Kinesis + Auto Loader), silver, gold hourly at :15, Power BI Pro refresh every 30 min31 to 90 min (avg 60)PullYes~$670Replace with P288% of the time is waiting for clocks: the hourly gold slot and Power BI's refresh timer.Faster platform lane, design A
P2Chained steps15-min run, gold rebuilt provisionally on every run (table-update trigger), gold job starts the Power BI import refresh6 to 22 min (avg 14)Pull (refresh started by a push signal)Yes~$770 + Power BI PPU, Premium or FabricPursue (recommended)Removes the two big waits for about $100 a month more; viewers read Power BI's in-memory copy, no idle queries.Faster platform lane, design B
P3Chained, every 5 minutesAs P2 with 5-minute runs5 to 10 minPullYes~$1,900Pursue only with a named need under 10 minAbout three times the platform-lane compute for a few minutes less.Faster platform lane, design C
P4Continuous pipelineAlways-on streaming tables, gold kept up to date, warm SQL warehouse, DirectQuery change detection every minute2 to 4 minPullYes~$2,400DropThree to four times the cost; if minutes are not enough, the need belongs in the fast lane.Faster platform lane, design D
P5P2 plus Lakehouse//RT servingAs P2; dashboards and apps query gold on a Lakehouse//RT warehouseas P2, queries in msPullYesP2 + beta pricing (30% off to Jan 2027)Pursue once out of beta, when many users are the painMillisecond queries for hundreds of users on governed gold; replaces ClickHouse for slicing. Does not make data fresher.Real time page, Lakehouse//RT
P6AWS onlyFirehose into S3 Tables (Iceberg), Glue jobs, Athena, QuickSight or Power BI via Athena15 to 75 minPullNo (Glue, Lake Formation)~$1,000 (whole stack)Pursue only without DatabricksFewest vendors; you build silver and gold jobs and the semantic layer yourself.Delta vs Iceberg
P7DirectQuery refreshing every minutePower BI queries the SQL warehouse on a 1-minute timer1 min after goldPullYeswarehouse never sleepsDropAbout 24,000 queries an hour for 50 viewers, almost all returning the same answer while gold changes hourly.Faster platform lane
P8Push rows from Databricks into Power BIGold job posts rows to a Power BI push datasetminutesPushNo-DropPush datasets are being retired, have limits, and duplicate the logic; push the signal (refresh), not the rows.Simulator, push or pull
P9File arrival trigger for irregular filesLab or partner file lands, Databricks file arrival trigger, Auto Loader reads every new fileabout 1 to 2 min to silverPull (checks each minute)Yes~$30PursueEvent-driven without glue code; a burst of files is one run; a missed event never loses a file.Simulator, what starts each step
P10One pipeline run per fileS3 event, Lambda, run the pipeline with the file path1 min + start-upPushYes-DropA start-up per file, runs queue, and passing paths skips Auto Loader's bookkeeping, so a missed event loses a file.Streaming guide, Databricks choices
P11Database changes as filesDMS (or Lakeflow Connect) writes change files to S3; the 15-min run reads them with Auto LoaderminutesPullYes~$60PursueNo event log needed until a live reader appears; SCD type 1 and 2 built in silver.Streaming guide, non-IoT

Verdict colours: green = pursue, blue = pursue under a condition, yellow = try first, rose = drop. Delays are for the production example (5 plants, 300 readings a second). Costs are rough US list prices from 2025 and 2026 for that lane only; see the cost calculator. "Screen gets data by" uses the rule: whoever starts the transfer; push if the sender starts it, pull if the screen or its server asks.

FAQ

Questions people ask

What is the recommended streaming design on AWS and Databricks?

Two lanes. The fast lane (F1): Kinesis, Managed Flink, DynamoDB and a WebSocket to a live board, 1 to 3 seconds, about $530 a month. The platform lane (P2): 15-minute Lakeflow runs with gold rebuilt on every run and the gold job starting the Power BI refresh, 6 to 22 minutes, about $770 a month.

When should the fast lane run on Databricks instead?

When having one platform governed end to end in Unity Catalog is worth about $200 to $600 a month more. Then use real-time mode in a serverless Lakeflow pipeline (F6), writing to Lakebase.

Does Lakehouse//RT replace the fast lane?

No. It answers queries in milliseconds but a screen is only as fresh as the table it reads. It fits as the serving layer on gold (P5).

Related

The detail pages

Streaming guide · Simulator: data flow · Simulator: three lanes · Simulator: a faster platform lane · Real-time mode and Lakehouse//RT · Delta vs Iceberg · Cost calculator