Home › Key concepts › Evaluation
Key concept · Walk (intermediate) · the same on every platform
Evaluation
Prove it works on cases you wrote down, every time you change it.
What it is
An evaluation set is a list of inputs with what a good result looks like. You score the system with rules in code where possible, a judge model where needed, and people for samples. Run it in CI so a worse version cannot ship.
Why it matters
Without evaluations, every prompt or model change is a guess, and you find out from customers.
An example
Fifty golden conversations run on every pull request; the new prompt drops grounding from 96% to 88%, so the merge is blocked.
The common mistake
Judging a change by trying three questions by hand.
Where you meet it on each platform
The idea stays the same; only the product name changes.
| Job | AWS | Azure | Google Cloud | Databricks | Open source |
|---|---|---|---|---|---|
| Evaluationprove it works on cases you wrote down, every release | Bedrock evaluations | Foundry evaluations | Vertex AI Gen AI evaluation | MLflow evaluation | Eval Harness Framework or promptfoo |
Related ideas
Where it shows up
Part of the Walk level on these paths: AI Engineering (safety and proof) · LLM Engineering (quality and data).