Home › Key concepts › Benchmarks and contamination

Key concept · Run (advanced) · the same on every platform

Benchmarks and contamination

Public scores are a starting point; your own tests decide.

What it is

Benchmarks are shared test sets for comparing models. Contamination is when test questions leaked into training data, inflating scores. Public scores rarely match your task, language or data.

Why it matters

Choosing a model by leaderboard alone is a common, costly mistake.

An example

A model ranked third publicly wins on your Hinglish support emails, where the top model struggles.

The common mistake

Never building a private test set of your own.

Where you meet it on each platform

The idea stays the same; only the product name changes.

JobAWSAzureGoogle CloudDatabricksOpen source
Evaluationprove it works on cases you wrote down, every releaseBedrock evaluationsFoundry evaluationsVertex AI Gen AI evaluationMLflow evaluationEval Harness Framework or promptfoo

Related ideas

Where it shows up

Part of the Run level on these paths: LLM Engineering (quality and data).