Home › Key concepts › Batch vs streaming

Key concept · Walk (intermediate) · the same on every platform

Batch vs streaming

Process data in scheduled chunks, or event by event as it happens.

What it is

Batch processing collects data and handles it together on a schedule, such as every night or every hour. Streaming handles each event within seconds of it happening. Streaming is fresher but harder: events arrive late, twice, or out of order.

BatchEvents pile upall dayNightly jobprocessesthe pileReportfresh by9 a.m.StreamEach eventas it happensStream jobhandles it insecondsAlert / tablefresh withinseconds
Batch is simpler and cheaper; streaming is fresher. Choose by how fresh the answer must be.

Why it matters

Choose by how fresh the answer must be. Most reports are fine with hourly batch; fraud checks and live stock levels are not.

An example

Nightly batch builds yesterday's sales report. A stream flags a card payment as suspicious within two seconds.

The common mistake

Building streaming because it sounds modern, when the business looks at the number once a day.

Where you meet it on each platform

The idea stays the same; only the product name changes.

JobAWSAzureGoogle CloudDatabricksOpen source
Streaminghandle events as they happen, not once a nightKinesis / Amazon MSKEvent Hubs / Fabric EventstreamPub/Sub + DataflowStructured StreamingKafka or Redpanda
Batch computeprocess a lot of data at onceAWS Glue / EMR (Spark)Fabric SparkDataproc (Spark)Databricks jobs (Spark)PySpark or Polars

Related ideas

Where it shows up

Part of the Walk level on these paths: Data Engineering (moving data).