Home › Key concepts › Training data: collect, clean, dedupe, licence
Key concept · Run (advanced) · the same on every platform
Training data: collect, clean, dedupe, licence
Good models start with good, legal, de-duplicated data.
What it is
Building a training set means collecting text, removing junk and personal data, filtering for quality, removing near-duplicates, and recording where each piece came from and under which licence. A data card documents all of it.
Why it matters
Data quality matters more than most model tricks, and licences matter for anything public.
An example
From 2 TB of scraped text, filtering and de-duplication keep 300 GB worth training on.
The common mistake
Training on data you cannot legally use, or cannot explain.
Where you meet it on each platform
The idea stays the same; only the product name changes.
| Job | AWS | Azure | Google Cloud | Databricks | Open source |
|---|---|---|---|---|---|
| Training data at scalecollect, clean, dedupe and filter text for models | EMR / Glue on S3 | Fabric Spark on OneLake | Dataflow / Dataproc on GCS | Spark on Delta | datatrove or Spark |
| Object storagecheap files in the cloud, the floor everything stands on | Amazon S3 | ADLS Gen2 / OneLake | Cloud Storage | Unity Catalog volumes on cloud storage | Local disk or MinIO |
Related ideas
Where it shows up
Part of the Run level on these paths: LLM Engineering (quality and data).