Home › Key concepts › Training-serving skew
Key concept · Walk (intermediate) · the same on every platform
Training-serving skew
When the model sees different data in production than it was trained on.
What it is
Skew happens when features are computed one way for training (a batch SQL query) and another way when serving (application code). Small differences, such as time zones or missing-value rules, quietly ruin predictions. A feature store computes them once for both.
Why it matters
It is one of the most common reasons a good model disappoints in production.
An example
Training used UTC dates and serving used local time, so 'orders today' was off for a third of customers.
The common mistake
Re-writing feature logic by hand for the serving path.
Where you meet it on each platform
The idea stays the same; only the product name changes.
| Job | AWS | Azure | Google Cloud | Databricks | Open source |
|---|---|---|---|---|---|
| Feature storethe same feature values for training and for live predictions | SageMaker Feature Store | Azure ML feature store | Vertex AI Feature Store | Feature engineering in Unity Catalog | Feast |
| Model servingan endpoint that answers predictions | SageMaker endpoints | Azure ML online endpoints | Vertex AI endpoints | Model Serving | FastAPI or BentoML |
Related ideas
Where it shows up
Part of the Walk level on these paths: ML Engineering (features and data).