Home › Key concepts › Training-serving skew

Key concept · Walk (intermediate) · the same on every platform

Training-serving skew

When the model sees different data in production than it was trained on.

What it is

Skew happens when features are computed one way for training (a batch SQL query) and another way when serving (application code). Small differences, such as time zones or missing-value rules, quietly ruin predictions. A feature store computes them once for both.

Why it matters

It is one of the most common reasons a good model disappoints in production.

An example

Training used UTC dates and serving used local time, so 'orders today' was off for a third of customers.

The common mistake

Re-writing feature logic by hand for the serving path.

Where you meet it on each platform

The idea stays the same; only the product name changes.

JobAWSAzureGoogle CloudDatabricksOpen source
Feature storethe same feature values for training and for live predictionsSageMaker Feature StoreAzure ML feature storeVertex AI Feature StoreFeature engineering in Unity CatalogFeast
Model servingan endpoint that answers predictionsSageMaker endpointsAzure ML online endpointsVertex AI endpointsModel ServingFastAPI or BentoML

Related ideas

Where it shows up

Part of the Walk level on these paths: ML Engineering (features and data).