Home › Data engineering › Streaming › Simulator

Data engineering · simulator · illustrative

Streaming simulator: a faster platform lane

The platform lane takes the same readings to Power BI through bronze, silver and gold. Today the screen can show data up to about 90 minutes old. Four designs run here on one clock, so you can see where the waiting goes and what removing it costs.

1 · Data flow, hop by hop2 · Three lanes together3 · A faster platform laneAll optionsThe streaming guide
Simulated time--:-- Each chart is the age of the newest data on the screen: it climbs while nothing new arrives and drops at each refresh.

Why today's lane is slow: where the time goes

Follow one reading from the press to the Power BI screen in design A. On average it takes about 60 minutes (best about 31, worst about 90). Only about 7 of those minutes are work. The rest is waiting for clocks.

StepAverageWorstShare of the averageWork or waiting?
Machine to Kinesis1 s1 s0%Work
Wait for the hour to close (gold is built per closed hour)30 min60 min50%Waiting: the gold design
Wait for the gold job's :15 slot (the 15-min run to silver happens inside this wait)15 min15 min25%Waiting: a clock
Gold job runs4 min5 min7%Work
Wait for Power BI's next 30-minute refresh11 min11 min18%Waiting: a clock
Totalabout 60 minabout 90 minabout 88% waiting

So three quarters of the time comes from building gold once an hour, and most of the rest from Power BI's refresh clock. The 15-minute run cadence, the obvious suspect, is hidden inside the hourly wait; it only starts to matter after gold is chained. In design B the same reading takes about 14 minutes on average: about 7.5 waiting for the next run, 3 for the run, 2 for gold and 1.5 for the refresh. Now the cadence is the biggest share, which is what design C shortens.

The architecture changes, one by one

ChangeProsConsVerdict
1 · Chain the steps instead of clocks. When the silver run finishes, a table update trigger (or the last task of the same job) starts gold; when gold finishes, a task calls the Power BI refresh API.Removes the two biggest waits; no new servicesA failed step must stop the chain and alertPursue
2 · Provisional gold every run. Each run recomputes the current hour's window, not only closed hours; every hour keeps a "complete" flag.Gold within minutes of silver; the same window-recompute codeA few more small gold runs a day; people must read the flagPursue
3 · Power BI capacity that can refresh often. Import model refreshed by the API after each gold run, or DirectQuery with change detection.Viewers read memory; no idle queriesPro allows only 8 refreshes a day; needs Premium Per User, Premium or FabricPursue with change 1
4 · Runs every 5 minutes.Screen 5 to 10 min oldAbout three times the platform-lane compute; more small files to compactPursue only if a named user needs under 10 minutes
5 · Continuous pipeline. Always-on streaming tables, gold kept up to date incrementally, a warm SQL warehouse, DirectQuery checks every minute.Screen 2 to 4 min oldThree to four times the cost; always-on compute to watchDrop: if minutes are not enough, the need belongs in the fast lane

Results at a glance

DesignScreen data age, best to worstRough platform cost a monthPower BI licenceVerdict
A · Todayabout 30 to 90 min~$670ProReplace
B · Chainedabout 6 to 22 min~$770PPU, Premium or FabricPursue (recommended)
C · Every 5 minutesabout 5 to 10 min~$1,900PPU, Premium or FabricOnly with a named need
D · Continuousabout 2 to 4 min~$2,400Premium or FabricDrop

Power BI licence in the diagram: Pro; PPU+ = Premium Per User, Premium or Fabric capacity; Premium = Premium or Fabric capacity. Rough US list prices, production example (300 readings a second); Power BI user licences not included. The simulator measures the age of the newest data on the screen; a single reading can be older if it arrived late. Faster lanes make numbers visible sooner, not final sooner: an hour is final only when its slowest input (lab results, nightly master data) has arrived.

Both run all day. Per CPU-hour they cost about the same; the difference is how small each can go. Flink scales down to one small processing unit. A DLT (Lakeflow Declarative Pipelines) pipeline is a Spark cluster with a driver and at least one worker, and on classic compute you pay the machines and the Databricks fee separately.

Managed Flink, continuousDLT continuous, classic computeDLT continuous, serverless
How it is billedAbout $0.11 per processing unit an hour (1 vCPU, 4 GB), plus 1 unit per app for orchestration, plus about $0.10 per GB a month of running stateEC2 machines, plus DBUs at about $0.20 (Core), $0.25 (Pro) or $0.36 (Advanced)One DBU price that includes the compute
Smallest setup1 + 1 units: about $160 a monthDriver + 1 worker (2 × m5.xlarge): EC2 about $280 + DBUs about $200 to $360 = about $480 to $650Not a fixed figure; expect about $600 to $1,500 for a small always-on pipeline (measure it)
Production example (300 readings a second)2 + 1 units: about $250Driver + 2 workers: about $700 to $1,000About $700 to $1,500
Stretch example (3,300 readings a second)4 to 8 units: about $450 to $7503 to 5 workers: about $1,000 to $1,800About $1,200 to $2,500
Per vCPU-hourAbout $0.11About $0.08 to $0.11 (EC2 + DBU)Higher: the compute is managed for you
DelayMillisecondsSeconds (micro-batches)Seconds
Ways to cut itFewer units; Lambda for simple limit alerts (about $8 a month)Graviton machines, committed-use discounts (spot workers are risky for an always-on job)Committed-use discounts
UseChoiceVerdict
Live lane: 1-minute windows and limit alerts into the live store in secondsManaged Flink, about $250 a monthPursue
Platform lane: bronze and silverDLT continuousDrop: the 15-minute pipeline (about $470) meets the minutes promise, and design B gets screens to 6 to 22 minutes
One engine for both lanes (no Flink)DLT continuous for the live lane tooDrop: about 2 to 4 times the cost, seconds instead of milliseconds, and a custom sink to the live store. If a second engine is the real problem, look at Spark real-time mode (fixed always-on classic cluster).

Rough US list prices from 2025 and 2026, not quotes. Check the AWS and Databricks pricing pages, or use the cost calculator.

What just happened

FAQ

Questions people ask

How do I make a Databricks to Power BI pipeline faster without streaming continuously?

Chain the steps instead of waiting for clocks: the 15-minute run starts gold, and gold starts the Power BI refresh through the API. The screen then shows data about 6 to 22 minutes old instead of 30 to 90, for little extra cost, but Power BI needs PPU, Premium or Fabric capacity for that many refreshes.

Is a continuous pipeline worth it for dashboards?

Rarely. It brings the screen to about 2 to 4 minutes for roughly three to four times the cost. If someone must act in seconds, that belongs in the fast lane; if minutes are fine, chained 15-minute runs are enough.

Is continuous Flink cheaper than continuous DLT?

Usually, for small streams. Managed Flink can run on one or two small processing units (about $160 to $250 a month); the smallest continuous DLT pipeline is a driver and a worker (about $480 to $650 on classic compute, more on serverless). Per CPU-hour they are similar, so the gap shrinks as volume grows.

Does a faster pipeline make an hour final sooner?

No. An hour is final only when its slowest input arrives (lab results, nightly master data). A faster lane shows provisional numbers sooner; each hour keeps a 'complete' flag.

Related

Where it connects

Seconds or minutes in the streaming guide, the cost calculator, and all simulators.