Home › Key concepts › Batch vs online prediction
Key concept · Walk (intermediate) · the same on every platform
Batch vs online prediction
Score everything on a schedule, or one request at a time as it comes.
What it is
Batch prediction scores many rows on a schedule and stores the results, which is cheap and simple. Online prediction answers each request in milliseconds through an endpoint. Use online only when the answer must reflect the last few seconds.
Why it matters
Online serving costs more and is harder to run. Many problems don't need it.
An example
Churn scores are computed nightly for all customers; fraud scores are computed live on each payment.
The common mistake
Building a real-time endpoint for a score that is read once a week.
Where you meet it on each platform
The idea stays the same; only the product name changes.
| Job | AWS | Azure | Google Cloud | Databricks | Open source |
|---|---|---|---|---|---|
| Model servingan endpoint that answers predictions | SageMaker endpoints | Azure ML online endpoints | Vertex AI endpoints | Model Serving | FastAPI or BentoML |
| ML pipelinesretrain, test and release a model on a schedule | SageMaker Pipelines | Azure ML pipelines | Vertex AI Pipelines | Lakeflow Jobs + MLflow | Metaflow or Kubeflow |
Related ideas
Where it shows up
Part of the Walk level on these paths: ML Engineering (production ml).