Home › Data engineering › Streaming › Simulator

Data engineering · simulator · illustrative

Streaming simulator: data flow, hop by hop

IoT readings and ordinary sources (app events, database changes, lab files) travel from the source to a live screen in seconds, and through bronze, silver and gold to a Power BI report in minutes. Pick a binding and a route, then break things.

1 · Data flow, hop by hop2 · Three lanes together3 · A faster platform laneAll optionsThe streaming guide
Binding
Route to the screen
Sources
Break something

Simulated time--:-- Dots move at viewing speed; the numbers run on simulated time.
Fast lane: reading to live screen-target ≤ 5 s
Platform lane: reading to silver-target ≤ 20 min
Platform lane: hour to gold report-target ≤ 75 min

Dots: filled circle = press reading, ring = connected tyre, triangle = app event, square = database change, diamond = lab file, green = a platform run moving rows, red cross = broken message.

Push or pull, chosen and dropped, hop by hop

Every arrow in the diagram carries a tag: green PUSH or blue PULL.

What starts each step: continuous, schedule or file event

Only the fast lane runs all the time. Everything in the platform lane runs, finishes and stops; what differs is what starts it.

StepStarted byHow it behavesVerdict
Fast lane (Flink, push to screens)Continuous, always onReads every record as it arrives; seconds behind.Pursue: someone acts within seconds
Kinesis records to bronze and silverSchedule: a Lakeflow job every 15 minStarts, reads everything new since its checkpoint ("available now"), writes bronze and silver, stops. Not continuous.Pursue: minutes are enough, and you pay only for run time
App event and database change filesSame 15-min scheduleFiles wait in S3; their file events wait in a queue; Auto Loader inside the run reads every new file once.Pursue
Lab result files (once an hour, at odd times)File event: Databricks file arrival triggerDatabricks checks the folder about once a minute and starts the job when new files appear; Auto Loader inside reads all of them. A burst of files is one run.Pursue for irregular drops
Gold hoursSchedule: hourly at :15Recomputes the hours that changed. Option: a table update trigger (start when silver changes) instead of the clock.Pursue the schedule; try the table trigger later
SAP and MES mastersSchedule: nightly 01:00Batch; refreshes dimensions and reference data for the fast lane.Pursue
Continuous streaming tables for the platform laneContinuousSilver about 1 minute behind, but a cluster runs all day.Drop: about $700 to $1,500 a month for minutes nobody needs
One pipeline run per file (S3 event, Lambda, run with the file path)File event, per fileA start-up per file; runs queue; a missed event loses a file.Drop

What to try

  1. Watch one 15-minute run. Stream readings pile up at lake ingest; at :00, :15, :30 and :45 a run moves them to bronze and silver. At :15 each hour, gold builds the previous hour and the report updates.
  2. Watch Kinesis. Each reading enters one shard, chosen by its press key, so one press stays in order. Two readers read the stream on their own: the live lane a couple of seconds behind, the lake reader up to 15 minutes behind until the next run (normal). Turn on connected tyres and the stream grows from 2 to 6 shards.
  3. Watch the files. App events (bundled by Firehose) and database changes (written by DMS) land as files in S3 and wait there; the 15-minute run reads them with Auto Loader, which remembers which files it has already loaded. A lab file is different: it lands once an hour at an odd time, so a Databricks file arrival trigger starts a load about a minute after it lands, instead of waiting for the next run. Either way the pipeline is never handed a single file; Auto Loader finds every new file itself.
  4. Switch the route to "Direct". The screen stays live, but database changes and lab files have nowhere to go, and no number is ever final.
  5. Switch to "Through the medallion". Every number is governed, but the screen is 15 to 75 minutes behind.
  6. Take a plant gateway offline. The live screen goes stale, the gateway buffers, and when it returns the late readings are counted and the gold hour is marked provisional.
  7. Change the binding. The roles stay the same; only the product in each box changes.

What it costs to run

Rough monthly list prices for the production example (5 plants, 300 readings a second), US regions, 2025 to 2026. They change with the binding, the route and the sources you turn on. Licences for Power BI users are not included. The full model, with every input, is in the cost calculator.

What just happened

FAQ

Questions people ask

Is this simulator showing real measured numbers?

No. It is an illustrative model of the design on the streaming guide: rates, run times and delays are typical values, not recordings.

Why does the platform lane wait for runs instead of streaming continuously?

Running every 15 minutes and stopping costs a fraction of an always-on cluster and meets a minutes-level promise. Only the fast lane runs all the time, and it only does light work.

Can Power BI show real-time data?

Power BI's old push and streaming datasets are being retired (no new ones after 31 Oct 2027). For seconds-level screens use a live web board fed by the fast lane; for reports, Power BI reads gold, refreshed when the gold job finishes.

Related

Where it connects

Streaming architecture on AWS and Databricks explains every box and option. Real-time analytics covers push to the screen; the source questionnaire (group O) lists what to ask a streaming source; all simulators.