D5 · Context & Reliability › Cluster 8 · Observability & Production Ops

8.4 · Evaluations in Production

Catch regressions & drift

CONCEPT Evals are your regression suite for prompts and models

Metrics (8.1–8.3) tell you the system is up; they don't tell you the answers are good. That's what evals are for. Build a held-out test set that mirrors your real task distribution — baseline cases, edge cases (typos, sarcasm, mixed sentiment), and adversarial inputs — then run it as a regression suite before every prompt or model change. Anthropic's guidance: prioritize volume over hand-crafted quality — many automatically-graded cases beat a handful of lovingly hand-graded ones.

Grading methodBest forCost / scale
Code-based (exact match, string, regex)Categorical answers — classification, extractionCheap, instant, fully automatable
LLM-as-judge (rubric-scored)Subjective dimensions — tone, coherence, faithfulnessScales to subjective grading; use a different model
Human gradingGround truth, complex judgment, validationSlow & costly — reserve for calibration
🔑 Make success criteria SMART: not "good sentiment analysis" but "F1 ≥ 0.85 on a held-out set of 10,000 tweets." Quantify even hazy goals — "<0.1% of outputs flagged for toxicity in 10,000 trials."

CONCEPT Sample live traffic to catch drift

A static test set goes stale as real inputs evolve. Periodically sample from production traffic and fold it back into the eval set so grades track reality — this is how you catch drift: a slow degradation as the input distribution shifts away from what you originally tested. For subjective quality at scale, run LLM-as-judge: a different model scores the primary model's output against an explicit rubric (Likert 1–5, binary yes/no, or ordinal). Using a separate grader model avoids a model rubber-stamping its own work.

Two triggers for a full eval run: (1) any prompt or model change → run the regression suite before shipping; (2) continuous → sample live traffic, grade it, watch the score trend for drift.

DIAGRAM Regression gate + drift watch

flowchart TD CH["Prompt / model change"] --> REG["Run regression eval
on held-out set"] REG --> G{"Scores hold
vs baseline?"} G -->|"yes"| SHIP["🟢 Ship"] G -->|"no"| BLOCK["🔴 Block — regression"] PROD["Live traffic"] -.->|"sample"| JUDGE["LLM-as-judge
(different model + rubric)"] JUDGE --> TREND{"Score trending
down?"} TREND -->|"yes"| DRIFT["🟠 Drift alert →
refresh eval set"] TREND -->|"no"| OK["🟢 Stable"] DRIFT -.->|"fold samples in"| REG classDef ok fill:#d4edda,stroke:#28a745,color:#155724; classDef gate fill:#ffe8cc,stroke:#ff9f40,color:#9a4a0a; classDef bad fill:#f8d7da,stroke:#dc3545,color:#721c24; class SHIP ok; class OK ok; class DRIFT gate; class BLOCK bad;

CODE LLM-as-judge with a different model + rubric

A binary faithfulness judge. The grader is a separate model scoring against an explicit rubric; aggregate pass-rate over the suite is your regression signal.

def judge_faithful(question, context, answer):
    rubric = f"""You are grading an answer for faithfulness to the source.
<context>{context}</context>
<answer>{answer}</answer>
Is every claim in the answer supported by the context?
Output only 'yes' or 'no'."""
    # Grade with a DIFFERENT model than the one under test.
    r = grader.messages.create(model="claude-opus-4-8", max_tokens=5,
                               messages=[{"role": "user", "content": rubric}])
    return r.content[0].text.strip().lower() == "yes"

# Regression gate: run the whole suite, compare to baseline before shipping.
passed = sum(judge_faithful(c.q, c.ctx, run_under_test(c)) for c in suite)
score = passed / len(suite)
assert score >= BASELINE - 0.02, f"regression: {score:.3f} < baseline"

TRADE-OFFS Code-based vs. LLM-as-judge grading

✅ Code-based grading

  • Deterministic, instant, free — run on every commit
  • Perfect for classification / extraction
  • No grader model to drift or bias

⚖️ LLM-as-judge

  • Grades tone, coherence, faithfulness at scale
  • But: needs a different grader model + clear rubric
  • Grader itself must be validated against human labels

EXAM TRAPS

Trap: Grading with the same model you're evaluating. → Best practice is a different model as judge, scored against an explicit rubric — a model judging its own output is biased.
Trap: Treating uptime/latency dashboards as "evals." → Those confirm the system is up. Evals confirm the answers are correct. You need both.
Trap: A frozen test set forever. → Input distributions drift. Sample live traffic periodically and fold it back in, or your green suite will slowly stop reflecting reality.
Trap: Shipping a prompt tweak without re-running evals. → Prompt and model changes are exactly when regressions sneak in — run the suite as a gate before shipping, not after users complain.

PRACTICE

Q1 · When should the regression eval suite run?

Before shipping any prompt or model change — those are exactly when regressions appear. The suite is a gate: block the change if scores drop below baseline. Additionally run it continuously on sampled live traffic to catch drift.

Q2 · Why use a different model for LLM-as-judge, and what else does it need?

A model grading its own output is biased toward passing itself, so use a separate grader model. It also needs an explicit rubric (binary, Likert 1–5, or ordinal) so grades are consistent and the judge can be validated against human labels.

Q3 · What is drift and how do production evals surface it?

Drift is gradual degradation as real inputs shift away from what you originally tested. Surface it by sampling live traffic, grading it (often via LLM-as-judge), and watching the score trend — a downward trend flags drift and tells you to refresh the eval set.