Home › Key concepts › Train, validation and test sets
Key concept · Crawl (beginner) · the same on every platform
Train, validation and test sets
Keep some data back so you can tell if the model really learned.
What it is
Train on one part of the data, tune settings on a second (validation), and report results once on a third (test) the model never saw. For data over time, split by date: train on the past, test on the future.
Why it matters
A model always looks good on data it has seen. Only held-out data tells the truth.
An example
Train on January–September, validate on October, test on November.
The common mistake
Shuffling time-series data randomly, so the model 'predicts' the past from the future.
Where you meet it on each platform
The idea stays the same; only the product name changes.
| Job | AWS | Azure | Google Cloud | Databricks | Open source |
|---|---|---|---|---|---|
| Trainingfit a model on data, on a laptop or a cluster | SageMaker training jobs | Azure ML jobs | Vertex AI training | Databricks ML runtime | scikit-learn, PyTorch |
| Experiment trackingremember which run, data and settings gave which score | SageMaker MLflow | Azure ML (MLflow) | Vertex AI Experiments | MLflow | MLflow |
Related ideas
Where it shows up
Part of the Crawl level on these paths: ML Engineering (learning basics).