Home › Data engineering › Streaming › All options
Data engineering · one table
Streaming options on AWS at a glance: fast lane and platform lane
Every way we looked at to get streaming data to a screen, in one table: 23 options, 4 recommended. Each row gives the whole path, how fresh the screen is, whether data is pushed or pulled to it, whether it sits in Unity Catalog, rough cost, the verdict and why, and a link to the details.
| # | Option | Path, hop by hop | Delay to the screen | Screen gets data by | In Unity Catalog | Rough cost a month | Verdict | Why | Details |
|---|---|---|---|---|---|---|---|---|---|
| Fast lane: seconds, for acting now | |||||||||
| F1 | AWS-native lane | Kinesis, Managed Flink (windows, limits), DynamoDB, DynamoDB Streams read by Lambda, API Gateway WebSocket, React live board | 1 to 3 s | Push | No (IAM) | ~$530 | Pursue (default) | Cheapest true push in milliseconds to seconds; serverless parts; the live store is a 48-hour copy, the record stays governed in gold. | Simulator, AWS + Databricks |
| F2 | F1 with AppSync Events | As F1, but the function publishes to an AppSync Events channel instead of tracking WebSocket connections | 1 to 3 s | Push | No | ~$500 | Try | Same lane with less code: AWS handles the fan-out to screens, no connections table. | Streaming guide, push and alerts |
| F3 | Alerts only, no Flink | Kinesis, Lambda limit checks, SNS alerts and a small live store | under 1 s | Push | No | ~$180 | Pursue if only limit alerts are needed | Nearly free, but no time windows, no drift detection, no state. | Streaming guide, live processor |
| F4 | Flink into ClickHouse | Kinesis, Flink (or ClickPipes), ClickHouse; the gateway polls every second and pushes differences | 2 to 5 s | Pull, then push | No | ~$750 to $3,500 | Pursue only if live screens need slicing beyond 48 hours | Fast ad-hoc filters for many users, but a new vendor, no push, and a third copy of the numbers. | Streaming guide, to the UI |
| F5 | Flink into Lakebase | Kinesis, Flink, Lakebase (Postgres in Databricks); the app polls Lakebase | 2 to 5 s | Pull, then push | Yes (store) | ~$450 to $650 | Pursue if the live store must be in Unity Catalog | Keeps the cheap Flink lane but moves the live store under Unity Catalog; loses free push from DynamoDB Streams. | Real time page, live store |
| F6 | Databricks real-time mode, serverless pipeline | Kinesis or Kafka, Lakeflow pipeline in real-time mode (GA 29 Oct 2026), custom writer into Lakebase, Databricks App, browser | 1 to 3 s | Pull, then push | Yes | ~$700 to $1,100 | Pursue if one governed platform is worth the extra cost | One engine and one platform, governed end to end, no cluster to size; costs more, cannot write Delta, the app polls. | Real time page |
| F7 | Databricks real-time mode, classic job | As F6, but a Structured Streaming job on a fixed classic cluster | 1 to 3 s | Pull, then push | Yes | ~$700 to $1,100 | Drop (prefer F6) | Same result as F6, but you size and run a fixed always-on cluster. | Real time page, two ways to run it |
| F8 | Lakehouse//RT as the fast lane | Pipelines into Delta, Lakehouse//RT warehouse, dashboard | minutes (table freshness) | Pull | Yes | beta pricing | Drop as a fast-lane replacement | It makes queries fast, not data fresh: a screen is only as fresh as the table it reads. | Real time page, Lakehouse//RT |
| F9 | Power BI streaming or push datasets | Processor pushes rows straight into Power BI | seconds | Push | No | - | Drop | Being retired (no new ones after 31 Oct 2027), rate-limited, and a second copy of the logic. | Simulator, push or pull |
| F10 | Push with no live store | Processor pushes straight to screens | about 1 s | Push | No | ~$100 | Drop | A screen that opens or reconnects has nothing to show until the next change; no last-hour chart. | Streaming guide, to the UI |
| F11 | Zerobus as the only entry | Devices straight into a bronze Delta table | seconds to bronze | - | Yes | - | Drop for sensors | One reader only: no shared event log for a live lane and the lake, no replay from the log. | Streaming guide, front door |
| F12 | Timestream LiveAnalytics, IoT Events | AWS time-series store and detector service | - | - | No | - | Drop | Timestream LiveAnalytics is closed to new customers; IoT Events reached end of support on 20 May 2026. | Streaming guide, notes |
| Platform lane: minutes, the number of record, to Power BI | |||||||||
| P1 | Today: hourly gold, Pro refresh | 15-min scheduled Lakeflow pipeline (Kinesis + Auto Loader), silver, gold hourly at :15, Power BI Pro refresh every 30 min | 31 to 90 min (avg 60) | Pull | Yes | ~$670 | Replace with P2 | 88% of the time is waiting for clocks: the hourly gold slot and Power BI's refresh timer. | Faster platform lane, design A |
| P2 | Chained steps | 15-min run, gold rebuilt provisionally on every run (table-update trigger), gold job starts the Power BI import refresh | 6 to 22 min (avg 14) | Pull (refresh started by a push signal) | Yes | ~$770 + Power BI PPU, Premium or Fabric | Pursue (recommended) | Removes the two big waits for about $100 a month more; viewers read Power BI's in-memory copy, no idle queries. | Faster platform lane, design B |
| P3 | Chained, every 5 minutes | As P2 with 5-minute runs | 5 to 10 min | Pull | Yes | ~$1,900 | Pursue only with a named need under 10 min | About three times the platform-lane compute for a few minutes less. | Faster platform lane, design C |
| P4 | Continuous pipeline | Always-on streaming tables, gold kept up to date, warm SQL warehouse, DirectQuery change detection every minute | 2 to 4 min | Pull | Yes | ~$2,400 | Drop | Three to four times the cost; if minutes are not enough, the need belongs in the fast lane. | Faster platform lane, design D |
| P5 | P2 plus Lakehouse//RT serving | As P2; dashboards and apps query gold on a Lakehouse//RT warehouse | as P2, queries in ms | Pull | Yes | P2 + beta pricing (30% off to Jan 2027) | Pursue once out of beta, when many users are the pain | Millisecond queries for hundreds of users on governed gold; replaces ClickHouse for slicing. Does not make data fresher. | Real time page, Lakehouse//RT |
| P6 | AWS only | Firehose into S3 Tables (Iceberg), Glue jobs, Athena, QuickSight or Power BI via Athena | 15 to 75 min | Pull | No (Glue, Lake Formation) | ~$1,000 (whole stack) | Pursue only without Databricks | Fewest vendors; you build silver and gold jobs and the semantic layer yourself. | Delta vs Iceberg |
| P7 | DirectQuery refreshing every minute | Power BI queries the SQL warehouse on a 1-minute timer | 1 min after gold | Pull | Yes | warehouse never sleeps | Drop | About 24,000 queries an hour for 50 viewers, almost all returning the same answer while gold changes hourly. | Faster platform lane |
| P8 | Push rows from Databricks into Power BI | Gold job posts rows to a Power BI push dataset | minutes | Push | No | - | Drop | Push datasets are being retired, have limits, and duplicate the logic; push the signal (refresh), not the rows. | Simulator, push or pull |
| P9 | File arrival trigger for irregular files | Lab or partner file lands, Databricks file arrival trigger, Auto Loader reads every new file | about 1 to 2 min to silver | Pull (checks each minute) | Yes | ~$30 | Pursue | Event-driven without glue code; a burst of files is one run; a missed event never loses a file. | Simulator, what starts each step |
| P10 | One pipeline run per file | S3 event, Lambda, run the pipeline with the file path | 1 min + start-up | Push | Yes | - | Drop | A start-up per file, runs queue, and passing paths skips Auto Loader's bookkeeping, so a missed event loses a file. | Streaming guide, Databricks choices |
| P11 | Database changes as files | DMS (or Lakeflow Connect) writes change files to S3; the 15-min run reads them with Auto Loader | minutes | Pull | Yes | ~$60 | Pursue | No event log needed until a live reader appears; SCD type 1 and 2 built in silver. | Streaming guide, non-IoT |
Verdict colours: green = pursue, blue = pursue under a condition, yellow = try first, rose = drop. Delays are for the production example (5 plants, 300 readings a second). Costs are rough US list prices from 2025 and 2026 for that lane only; see the cost calculator. "Screen gets data by" uses the rule: whoever starts the transfer; push if the sender starts it, pull if the screen or its server asks.
FAQ
Questions people ask
What is the recommended streaming design on AWS and Databricks?
Two lanes. The fast lane (F1): Kinesis, Managed Flink, DynamoDB and a WebSocket to a live board, 1 to 3 seconds, about $530 a month. The platform lane (P2): 15-minute Lakeflow runs with gold rebuilt on every run and the gold job starting the Power BI refresh, 6 to 22 minutes, about $770 a month.
When should the fast lane run on Databricks instead?
When having one platform governed end to end in Unity Catalog is worth about $200 to $600 a month more. Then use real-time mode in a serverless Lakeflow pipeline (F6), writing to Lakebase.
Does Lakehouse//RT replace the fast lane?
No. It answers queries in milliseconds but a screen is only as fresh as the table it reads. It fits as the serving layer on gold (P5).
Related
The detail pages
Streaming guide · Simulator: data flow · Simulator: three lanes · Simulator: a faster platform lane · Real-time mode and Lakehouse//RT · Delta vs Iceberg · Cost calculator