CONCEPT Evals are your regression suite for prompts and models
Metrics (8.1–8.3) tell you the system is up; they don't tell you the answers are good. That's what evals are for. Build a held-out test set that mirrors your real task distribution — baseline cases, edge cases (typos, sarcasm, mixed sentiment), and adversarial inputs — then run it as a regression suite before every prompt or model change. Anthropic's guidance: prioritize volume over hand-crafted quality — many automatically-graded cases beat a handful of lovingly hand-graded ones.
| Grading method | Best for | Cost / scale |
|---|---|---|
| Code-based (exact match, string, regex) | Categorical answers — classification, extraction | Cheap, instant, fully automatable |
| LLM-as-judge (rubric-scored) | Subjective dimensions — tone, coherence, faithfulness | Scales to subjective grading; use a different model |
| Human grading | Ground truth, complex judgment, validation | Slow & costly — reserve for calibration |
CONCEPT Sample live traffic to catch drift
A static test set goes stale as real inputs evolve. Periodically sample from production traffic and fold it back into the eval set so grades track reality — this is how you catch drift: a slow degradation as the input distribution shifts away from what you originally tested. For subjective quality at scale, run LLM-as-judge: a different model scores the primary model's output against an explicit rubric (Likert 1–5, binary yes/no, or ordinal). Using a separate grader model avoids a model rubber-stamping its own work.
DIAGRAM Regression gate + drift watch
on held-out set"] REG --> G{"Scores hold
vs baseline?"} G -->|"yes"| SHIP["🟢 Ship"] G -->|"no"| BLOCK["🔴 Block — regression"] PROD["Live traffic"] -.->|"sample"| JUDGE["LLM-as-judge
(different model + rubric)"] JUDGE --> TREND{"Score trending
down?"} TREND -->|"yes"| DRIFT["🟠 Drift alert →
refresh eval set"] TREND -->|"no"| OK["🟢 Stable"] DRIFT -.->|"fold samples in"| REG classDef ok fill:#d4edda,stroke:#28a745,color:#155724; classDef gate fill:#ffe8cc,stroke:#ff9f40,color:#9a4a0a; classDef bad fill:#f8d7da,stroke:#dc3545,color:#721c24; class SHIP ok; class OK ok; class DRIFT gate; class BLOCK bad;
CODE LLM-as-judge with a different model + rubric
A binary faithfulness judge. The grader is a separate model scoring against an explicit rubric; aggregate pass-rate over the suite is your regression signal.
def judge_faithful(question, context, answer): rubric = f"""You are grading an answer for faithfulness to the source. <context>{context}</context> <answer>{answer}</answer> Is every claim in the answer supported by the context? Output only 'yes' or 'no'.""" # Grade with a DIFFERENT model than the one under test. r = grader.messages.create(model="claude-opus-4-8", max_tokens=5, messages=[{"role": "user", "content": rubric}]) return r.content[0].text.strip().lower() == "yes" # Regression gate: run the whole suite, compare to baseline before shipping. passed = sum(judge_faithful(c.q, c.ctx, run_under_test(c)) for c in suite) score = passed / len(suite) assert score >= BASELINE - 0.02, f"regression: {score:.3f} < baseline"
TRADE-OFFS Code-based vs. LLM-as-judge grading
✅ Code-based grading
- Deterministic, instant, free — run on every commit
- Perfect for classification / extraction
- No grader model to drift or bias
⚖️ LLM-as-judge
- Grades tone, coherence, faithfulness at scale
- But: needs a different grader model + clear rubric
- Grader itself must be validated against human labels
EXAM TRAPS
PRACTICE
Q1 · When should the regression eval suite run?
Before shipping any prompt or model change — those are exactly when regressions appear. The suite is a gate: block the change if scores drop below baseline. Additionally run it continuously on sampled live traffic to catch drift.
Q2 · Why use a different model for LLM-as-judge, and what else does it need?
A model grading its own output is biased toward passing itself, so use a separate grader model. It also needs an explicit rubric (binary, Likert 1–5, or ordinal) so grades are consistent and the judge can be validated against human labels.
Q3 · What is drift and how do production evals surface it?
Drift is gradual degradation as real inputs shift away from what you originally tested. Surface it by sampling live traffic, grading it (often via LLM-as-judge), and watching the score trend — a downward trend flags drift and tells you to refresh the eval set.