Home › Key concepts › Benchmarks and contamination
Key concept · Run (advanced) · the same on every platform
Benchmarks and contamination
Public scores are a starting point; your own tests decide.
What it is
Benchmarks are shared test sets for comparing models. Contamination is when test questions leaked into training data, inflating scores. Public scores rarely match your task, language or data.
Why it matters
Choosing a model by leaderboard alone is a common, costly mistake.
An example
A model ranked third publicly wins on your Hinglish support emails, where the top model struggles.
The common mistake
Never building a private test set of your own.
Where you meet it on each platform
The idea stays the same; only the product name changes.
| Job | AWS | Azure | Google Cloud | Databricks | Open source |
|---|---|---|---|---|---|
| Evaluationprove it works on cases you wrote down, every release | Bedrock evaluations | Foundry evaluations | Vertex AI Gen AI evaluation | MLflow evaluation | Eval Harness Framework or promptfoo |
Related ideas
Where it shows up
Part of the Run level on these paths: LLM Engineering (quality and data).