Home › Key concepts › Batch vs online prediction

Key concept · Walk (intermediate) · the same on every platform

Batch vs online prediction

Score everything on a schedule, or one request at a time as it comes.

What it is

Batch prediction scores many rows on a schedule and stores the results, which is cheap and simple. Online prediction answers each request in milliseconds through an endpoint. Use online only when the answer must reflect the last few seconds.

Why it matters

Online serving costs more and is harder to run. Many problems don't need it.

An example

Churn scores are computed nightly for all customers; fraud scores are computed live on each payment.

The common mistake

Building a real-time endpoint for a score that is read once a week.

Where you meet it on each platform

The idea stays the same; only the product name changes.

JobAWSAzureGoogle CloudDatabricksOpen source
Model servingan endpoint that answers predictionsSageMaker endpointsAzure ML online endpointsVertex AI endpointsModel ServingFastAPI or BentoML
ML pipelinesretrain, test and release a model on a scheduleSageMaker PipelinesAzure ML pipelinesVertex AI PipelinesLakeflow Jobs + MLflowMetaflow or Kubeflow

Related ideas

Where it shows up

Part of the Walk level on these paths: ML Engineering (production ml).