Home › Key concepts › Batch vs streaming
Key concept · Walk (intermediate) · the same on every platform
Batch vs streaming
Process data in scheduled chunks, or event by event as it happens.
What it is
Batch processing collects data and handles it together on a schedule, such as every night or every hour. Streaming handles each event within seconds of it happening. Streaming is fresher but harder: events arrive late, twice, or out of order.
Why it matters
Choose by how fresh the answer must be. Most reports are fine with hourly batch; fraud checks and live stock levels are not.
An example
Nightly batch builds yesterday's sales report. A stream flags a card payment as suspicious within two seconds.
The common mistake
Building streaming because it sounds modern, when the business looks at the number once a day.
Where you meet it on each platform
The idea stays the same; only the product name changes.
| Job | AWS | Azure | Google Cloud | Databricks | Open source |
|---|---|---|---|---|---|
| Streaminghandle events as they happen, not once a night | Kinesis / Amazon MSK | Event Hubs / Fabric Eventstream | Pub/Sub + Dataflow | Structured Streaming | Kafka or Redpanda |
| Batch computeprocess a lot of data at once | AWS Glue / EMR (Spark) | Fabric Spark | Dataproc (Spark) | Databricks jobs (Spark) | PySpark or Polars |
Related ideas
Where it shows up
Part of the Walk level on these paths: Data Engineering (moving data).