Home › Key concepts › Evaluation

Key concept · Walk (intermediate) · the same on every platform

Evaluation

Prove it works on cases you wrote down, every time you change it.

What it is

An evaluation set is a list of inputs with what a good result looks like. You score the system with rules in code where possible, a judge model where needed, and people for samples. Run it in CI so a worse version cannot ship.

Why it matters

Without evaluations, every prompt or model change is a guess, and you find out from customers.

An example

Fifty golden conversations run on every pull request; the new prompt drops grounding from 96% to 88%, so the merge is blocked.

The common mistake

Judging a change by trying three questions by hand.

Where you meet it on each platform

The idea stays the same; only the product name changes.

JobAWSAzureGoogle CloudDatabricksOpen source
Evaluationprove it works on cases you wrote down, every releaseBedrock evaluationsFoundry evaluationsVertex AI Gen AI evaluationMLflow evaluationEval Harness Framework or promptfoo

Related ideas

Where it shows up

Part of the Walk level on these paths: AI Engineering (safety and proof) · LLM Engineering (quality and data).