Home › Data engineering › Streaming › Simulator
Data engineering · simulator · illustrative
Streaming simulator: data flow, hop by hop
IoT readings and ordinary sources (app events, database changes, lab files) travel from the source to a live screen in seconds, and through bronze, silver and gold to a Power BI report in minutes. Pick a binding and a route, then break things.
Dots: filled circle = press reading, ring = connected tyre, triangle = app event, square = database change, diamond = lab file, green = a platform run moving rows, red cross = broken message.
Push or pull, chosen and dropped, hop by hop
Every arrow in the diagram carries a tag: green PUSH or blue PULL.
What starts each step: continuous, schedule or file event
Only the fast lane runs all the time. Everything in the platform lane runs, finishes and stops; what differs is what starts it.
| Step | Started by | How it behaves | Verdict |
|---|---|---|---|
| Fast lane (Flink, push to screens) | Continuous, always on | Reads every record as it arrives; seconds behind. | Pursue: someone acts within seconds |
| Kinesis records to bronze and silver | Schedule: a Lakeflow job every 15 min | Starts, reads everything new since its checkpoint ("available now"), writes bronze and silver, stops. Not continuous. | Pursue: minutes are enough, and you pay only for run time |
| App event and database change files | Same 15-min schedule | Files wait in S3; their file events wait in a queue; Auto Loader inside the run reads every new file once. | Pursue |
| Lab result files (once an hour, at odd times) | File event: Databricks file arrival trigger | Databricks checks the folder about once a minute and starts the job when new files appear; Auto Loader inside reads all of them. A burst of files is one run. | Pursue for irregular drops |
| Gold hours | Schedule: hourly at :15 | Recomputes the hours that changed. Option: a table update trigger (start when silver changes) instead of the clock. | Pursue the schedule; try the table trigger later |
| SAP and MES masters | Schedule: nightly 01:00 | Batch; refreshes dimensions and reference data for the fast lane. | Pursue |
| Continuous streaming tables for the platform lane | Continuous | Silver about 1 minute behind, but a cluster runs all day. | Drop: about $700 to $1,500 a month for minutes nobody needs |
| One pipeline run per file (S3 event, Lambda, run with the file path) | File event, per file | A start-up per file; runs queue; a missed event loses a file. | Drop |
What to try
- Watch one 15-minute run. Stream readings pile up at lake ingest; at :00, :15, :30 and :45 a run moves them to bronze and silver. At :15 each hour, gold builds the previous hour and the report updates.
- Watch Kinesis. Each reading enters one shard, chosen by its press key, so one press stays in order. Two readers read the stream on their own: the live lane a couple of seconds behind, the lake reader up to 15 minutes behind until the next run (normal). Turn on connected tyres and the stream grows from 2 to 6 shards.
- Watch the files. App events (bundled by Firehose) and database changes (written by DMS) land as files in S3 and wait there; the 15-minute run reads them with Auto Loader, which remembers which files it has already loaded. A lab file is different: it lands once an hour at an odd time, so a Databricks file arrival trigger starts a load about a minute after it lands, instead of waiting for the next run. Either way the pipeline is never handed a single file; Auto Loader finds every new file itself.
- Switch the route to "Direct". The screen stays live, but database changes and lab files have nowhere to go, and no number is ever final.
- Switch to "Through the medallion". Every number is governed, but the screen is 15 to 75 minutes behind.
- Take a plant gateway offline. The live screen goes stale, the gateway buffers, and when it returns the late readings are counted and the gold hour is marked provisional.
- Change the binding. The roles stay the same; only the product in each box changes.
What it costs to run
Rough monthly list prices for the production example (5 plants, 300 readings a second), US regions, 2025 to 2026. They change with the binding, the route and the sources you turn on. Licences for Power BI users are not included. The full model, with every input, is in the cost calculator.
What just happened
FAQ
Questions people ask
Is this simulator showing real measured numbers?
No. It is an illustrative model of the design on the streaming guide: rates, run times and delays are typical values, not recordings.
Why does the platform lane wait for runs instead of streaming continuously?
Running every 15 minutes and stopping costs a fraction of an always-on cluster and meets a minutes-level promise. Only the fast lane runs all the time, and it only does light work.
Can Power BI show real-time data?
Power BI's old push and streaming datasets are being retired (no new ones after 31 Oct 2027). For seconds-level screens use a live web board fed by the fast lane; for reports, Power BI reads gold, refreshed when the gold job finishes.
Related
Where it connects
Streaming architecture on AWS and Databricks explains every box and option. Real-time analytics covers push to the screen; the source questionnaire (group O) lists what to ask a streaming source; all simulators.