Home › Data engineering › Streaming › Simulator
Data engineering · simulator · illustrative
Streaming simulator: a faster platform lane
The platform lane takes the same readings to Power BI through bronze, silver and gold. Today the screen can show data up to about 90 minutes old. Four designs run here on one clock, so you can see where the waiting goes and what removing it costs.
Why today's lane is slow: where the time goes
Follow one reading from the press to the Power BI screen in design A. On average it takes about 60 minutes (best about 31, worst about 90). Only about 7 of those minutes are work. The rest is waiting for clocks.
| Step | Average | Worst | Share of the average | Work or waiting? |
|---|---|---|---|---|
| Machine to Kinesis | 1 s | 1 s | 0% | Work |
| Wait for the hour to close (gold is built per closed hour) | 30 min | 60 min | 50% | Waiting: the gold design |
| Wait for the gold job's :15 slot (the 15-min run to silver happens inside this wait) | 15 min | 15 min | 25% | Waiting: a clock |
| Gold job runs | 4 min | 5 min | 7% | Work |
| Wait for Power BI's next 30-minute refresh | 11 min | 11 min | 18% | Waiting: a clock |
| Total | about 60 min | about 90 min | about 88% waiting |
So three quarters of the time comes from building gold once an hour, and most of the rest from Power BI's refresh clock. The 15-minute run cadence, the obvious suspect, is hidden inside the hourly wait; it only starts to matter after gold is chained. In design B the same reading takes about 14 minutes on average: about 7.5 waiting for the next run, 3 for the run, 2 for gold and 1.5 for the refresh. Now the cadence is the biggest share, which is what design C shortens.
The architecture changes, one by one
| Change | Pros | Cons | Verdict |
|---|---|---|---|
| 1 · Chain the steps instead of clocks. When the silver run finishes, a table update trigger (or the last task of the same job) starts gold; when gold finishes, a task calls the Power BI refresh API. | Removes the two biggest waits; no new services | A failed step must stop the chain and alert | Pursue |
| 2 · Provisional gold every run. Each run recomputes the current hour's window, not only closed hours; every hour keeps a "complete" flag. | Gold within minutes of silver; the same window-recompute code | A few more small gold runs a day; people must read the flag | Pursue |
| 3 · Power BI capacity that can refresh often. Import model refreshed by the API after each gold run, or DirectQuery with change detection. | Viewers read memory; no idle queries | Pro allows only 8 refreshes a day; needs Premium Per User, Premium or Fabric | Pursue with change 1 |
| 4 · Runs every 5 minutes. | Screen 5 to 10 min old | About three times the platform-lane compute; more small files to compact | Pursue only if a named user needs under 10 minutes |
| 5 · Continuous pipeline. Always-on streaming tables, gold kept up to date incrementally, a warm SQL warehouse, DirectQuery checks every minute. | Screen 2 to 4 min old | Three to four times the cost; always-on compute to watch | Drop: if minutes are not enough, the need belongs in the fast lane |
Results at a glance
| Design | Screen data age, best to worst | Rough platform cost a month | Power BI licence | Verdict |
|---|---|---|---|---|
| A · Today | about 30 to 90 min | ~$670 | Pro | Replace |
| B · Chained | about 6 to 22 min | ~$770 | PPU, Premium or Fabric | Pursue (recommended) |
| C · Every 5 minutes | about 5 to 10 min | ~$1,900 | PPU, Premium or Fabric | Only with a named need |
| D · Continuous | about 2 to 4 min | ~$2,400 | Premium or Fabric | Drop |
Power BI licence in the diagram: Pro; PPU+ = Premium Per User, Premium or Fabric capacity; Premium = Premium or Fabric capacity. Rough US list prices, production example (300 readings a second); Power BI user licences not included. The simulator measures the age of the newest data on the screen; a single reading can be older if it arrived late. Faster lanes make numbers visible sooner, not final sooner: an hour is final only when its slowest input (lab results, nightly master data) has arrived.
Continuous Flink or continuous DLT: what each costs
Both run all day. Per CPU-hour they cost about the same; the difference is how small each can go. Flink scales down to one small processing unit. A DLT (Lakeflow Declarative Pipelines) pipeline is a Spark cluster with a driver and at least one worker, and on classic compute you pay the machines and the Databricks fee separately.
| Managed Flink, continuous | DLT continuous, classic compute | DLT continuous, serverless | |
|---|---|---|---|
| How it is billed | About $0.11 per processing unit an hour (1 vCPU, 4 GB), plus 1 unit per app for orchestration, plus about $0.10 per GB a month of running state | EC2 machines, plus DBUs at about $0.20 (Core), $0.25 (Pro) or $0.36 (Advanced) | One DBU price that includes the compute |
| Smallest setup | 1 + 1 units: about $160 a month | Driver + 1 worker (2 × m5.xlarge): EC2 about $280 + DBUs about $200 to $360 = about $480 to $650 | Not a fixed figure; expect about $600 to $1,500 for a small always-on pipeline (measure it) |
| Production example (300 readings a second) | 2 + 1 units: about $250 | Driver + 2 workers: about $700 to $1,000 | About $700 to $1,500 |
| Stretch example (3,300 readings a second) | 4 to 8 units: about $450 to $750 | 3 to 5 workers: about $1,000 to $1,800 | About $1,200 to $2,500 |
| Per vCPU-hour | About $0.11 | About $0.08 to $0.11 (EC2 + DBU) | Higher: the compute is managed for you |
| Delay | Milliseconds | Seconds (micro-batches) | Seconds |
| Ways to cut it | Fewer units; Lambda for simple limit alerts (about $8 a month) | Graviton machines, committed-use discounts (spot workers are risky for an always-on job) | Committed-use discounts |
| Use | Choice | Verdict |
|---|---|---|
| Live lane: 1-minute windows and limit alerts into the live store in seconds | Managed Flink, about $250 a month | Pursue |
| Platform lane: bronze and silver | DLT continuous | Drop: the 15-minute pipeline (about $470) meets the minutes promise, and design B gets screens to 6 to 22 minutes |
| One engine for both lanes (no Flink) | DLT continuous for the live lane too | Drop: about 2 to 4 times the cost, seconds instead of milliseconds, and a custom sink to the live store. If a second engine is the real problem, look at Spark real-time mode (fixed always-on classic cluster). |
Rough US list prices from 2025 and 2026, not quotes. Check the AWS and Databricks pricing pages, or use the cost calculator.
What just happened
FAQ
Questions people ask
How do I make a Databricks to Power BI pipeline faster without streaming continuously?
Chain the steps instead of waiting for clocks: the 15-minute run starts gold, and gold starts the Power BI refresh through the API. The screen then shows data about 6 to 22 minutes old instead of 30 to 90, for little extra cost, but Power BI needs PPU, Premium or Fabric capacity for that many refreshes.
Is a continuous pipeline worth it for dashboards?
Rarely. It brings the screen to about 2 to 4 minutes for roughly three to four times the cost. If someone must act in seconds, that belongs in the fast lane; if minutes are fine, chained 15-minute runs are enough.
Is continuous Flink cheaper than continuous DLT?
Usually, for small streams. Managed Flink can run on one or two small processing units (about $160 to $250 a month); the smallest continuous DLT pipeline is a driver and a worker (about $480 to $650 on classic compute, more on serverless). Per CPU-hour they are similar, so the gap shrinks as volume grows.
Does a faster pipeline make an hour final sooner?
No. An hour is final only when its slowest input arrives (lab results, nightly master data). A faster lane shows provisional numbers sooner; each hour keeps a 'complete' flag.
Related
Where it connects
Seconds or minutes in the streaming guide, the cost calculator, and all simulators.