Home › Key concepts › Training data: collect, clean, dedupe, licence

Key concept · Run (advanced) · the same on every platform

Training data: collect, clean, dedupe, licence

Good models start with good, legal, de-duplicated data.

What it is

Building a training set means collecting text, removing junk and personal data, filtering for quality, removing near-duplicates, and recording where each piece came from and under which licence. A data card documents all of it.

Why it matters

Data quality matters more than most model tricks, and licences matter for anything public.

An example

From 2 TB of scraped text, filtering and de-duplication keep 300 GB worth training on.

The common mistake

Training on data you cannot legally use, or cannot explain.

Where you meet it on each platform

The idea stays the same; only the product name changes.

JobAWSAzureGoogle CloudDatabricksOpen source
Training data at scalecollect, clean, dedupe and filter text for modelsEMR / Glue on S3Fabric Spark on OneLakeDataflow / Dataproc on GCSSpark on Deltadatatrove or Spark
Object storagecheap files in the cloud, the floor everything stands onAmazon S3ADLS Gen2 / OneLakeCloud StorageUnity Catalog volumes on cloud storageLocal disk or MinIO

Related ideas

Where it shows up

Part of the Run level on these paths: LLM Engineering (quality and data).