Home › Key concepts › Train, validation and test sets

Key concept · Crawl (beginner) · the same on every platform

Train, validation and test sets

Keep some data back so you can tell if the model really learned.

What it is

Train on one part of the data, tune settings on a second (validation), and report results once on a third (test) the model never saw. For data over time, split by date: train on the past, test on the future.

Why it matters

A model always looks good on data it has seen. Only held-out data tells the truth.

An example

Train on January–September, validate on October, test on November.

The common mistake

Shuffling time-series data randomly, so the model 'predicts' the past from the future.

Where you meet it on each platform

The idea stays the same; only the product name changes.

JobAWSAzureGoogle CloudDatabricksOpen source
Trainingfit a model on data, on a laptop or a clusterSageMaker training jobsAzure ML jobsVertex AI trainingDatabricks ML runtimescikit-learn, PyTorch
Experiment trackingremember which run, data and settings gave which scoreSageMaker MLflowAzure ML (MLflow)Vertex AI ExperimentsMLflowMLflow

Related ideas

Where it shows up

Part of the Crawl level on these paths: ML Engineering (learning basics).