Home › Data engineering › Streaming › Simulator

Data engineering · simulator · illustrative

Streaming simulator: three lanes together

One simulated day at a tyre plant. Watch the fast lane, the 15-minute near-real-time lane and the nightly batch run side by side, meet in gold, and land on three screens with three freshness promises.

1 · Data flow, hop by hop2 · Three lanes together3 · A faster platform laneAll optionsThe streaming guide
Binding
Route to the screen
Break something

Simulated time--:-- Dots move at viewing speed; the numbers run on simulated time.

Gold hours: green = complete, yellow = provisional, waiting for an input (lab = lab results, dims = nightly dimensions, gap = plant readings), blue = restated (recomputed after late data), dashed = the hour still running.

How the three lanes meet

  1. At the source. The fast lane and the platform lane each read the same raw events from the event log. Neither reads the other's output.
  2. Reference data, upward. Limits and the press list are copied from silver and gold into the fast lane on a schedule. The fast lane never calls the platform. Watch new press P-1501 show as "unknown press" until the nightly batch brings it into the press list.
  3. In gold. Near real time and batch are the same tables at different cadences. Hourly sensor facts join nightly dimensions and lab results that arrive hours later, so a gold hour is only as fresh as its slowest input.
  4. On the screen. The live screen, the hourly report and the daily report sit side by side. They show one dataset at three speeds, and they are never added together.

What to try

  1. Let a few hours pass. Each hour turns provisional at :15 and complete about two hours later, when its lab results land.
  2. Fail tonight's batch. Dimensions go stale, hours stay provisional, the daily report is late, and the new press stays unknown. The next night fixes all of it, and the waiting hours are restated.
  3. Take the plant offline for an hour. The live screen goes stale. When the buffer arrives, that gold hour is restated.

What it costs to run

Rough monthly list prices for the production example (5 plants, 300 readings a second), US regions, 2025 to 2026. They change with the binding, the route and the sources you turn on. Licences for Power BI users are not included. The full model, with every input, is in the cost calculator.

What just happened

FAQ

Questions people ask

Do near real time and batch need separate pipelines?

No. In this design they are the same pipeline and the same tables, run at different cadences: streams every 15 minutes, master data nightly. Only the seconds lane is separate.

Where do the fast lane and the batch lane meet?

In four places: the shared raw event log, reference data copied up into the fast lane, gold (where streams join batch dimensions) and the screen, where live and final numbers sit side by side.

Can one dataset be in all three speeds?

Yes: press readings appear on the live screen in seconds, in hourly gold in minutes and in the daily report the next morning. Only bronze, silver and gold are the record; the live copy expires.

Related

Where it connects

Streaming architecture on AWS and Databricks explains every box and option. Real-time analytics covers push to the screen; the source questionnaire (group O) lists what to ask a streaming source; all simulators.