CCA-P start here · Trap gym · Trap 9

Evidence before you ship

D4 · Evaluation, Testing & OptimizationGuide lines 4.3, 6.5

At a glance: A few good-looking examples, or a test that changed two things at once, is not evidence: run the eval set first, change one thing at a time, then pick the live test that fits the question.

When someone rewrites a prompt and tries it on ten conversations, the answers often look better. That feeling is an anecdote: ten hand-picked cases cannot show whether the change broke something on the other hundreds of kinds of question. The cheap, safe first step is the offline eval: run the old and new versions on the same full test set and compare the scores.

Once the offline eval passes, the next step depends on the question being asked. If the question is about real user behaviour (do more customers finish, do fewer call back), you need an A/B test: a random share of live traffic gets the new version, the rest keep the old one, over the same period. If traffic is too low for an A/B test to reach a clear answer, roll out in stages with close monitoring and a rollback. If showing users an unapproved answer is not allowed, run it in shadow: live inputs, outputs compared but never shown.

Two rules sit under all of this. Change one thing at a time, or nobody can say which change caused the result. And the same discipline applies to new model versions: a newer model is adopted through the eval suite and a staged rollout, not because it is newer.

Change one fact

A letting agency's tenant assistant answers questions about leases and repairs, about 3,000 conversations a day. An engineer has rewritten the system prompt, and on ten conversations she tried, the answers read better. The team keeps an offline eval set of 400 past questions with expected answers. What should the team do next with the new prompt?

Words in the stem that signal this trap

Rule to remember: Evidence comes in order: the offline eval first, then the live test that fits the question (A/B for behaviour, shadow when exposure is forbidden, staged rollout when traffic is low). Change one thing at a time.

Practice

Practice 9.1 · Choose one.

A legal publisher's research assistant lost five points on its eval suite after one release that changed both the retrieval chunk size and the answer prompt. Users have not noticed yet. The team wants to keep whichever change helped and drop the one that hurt. What should the team do?

  1. Revert the whole release, then test each change on its own against the eval suite.
  2. Keep the release live and add eval cases until the cause of the drop becomes clear.
  3. Revert only the prompt change, as prompt edits are the usual cause of quality drops.
  4. A/B test the full release against the previous version on a share of live traffic.
Show answer

Answer: A. The deciding fact: two things changed at once, and the team wants to know which one hurt. Reverting stops the regression, and testing each change alone shows which to keep.

  • A ✅ Revert the whole release, then test each change on its own against the eval suite.
  • B ❌ Works, but it leaves a known drop live, and more cases still cannot separate two changes made together.
  • C ❌ Works if the prompt was the cause, but that is a guess; the chunk size may be what hurt.
  • D ❌ Works for measuring user impact, but it still tests both changes together and exposes users to the drop.
Practice 9.2 · Choose one.

A bank's card-support assistant is moving to a newer model version, which scored better on the full eval suite. The suite was built two years ago from English conversations. Since then, a third of conversations arrive in Spanish, and no eval case is in Spanish. What should the team do before moving traffic?

  1. Begin a staged rollout now, since the suite already shows the new version is better.
  2. Run an A/B test on live traffic to see how Spanish-speaking customers respond.
  3. Add Spanish cases taken from real conversations to the suite, then rerun both versions.
  4. Ask a few Spanish-speaking staff to try the new version and report their impressions.
Show answer

Answer: C. The deciding fact: a third of real traffic is in a language the suite never tests, so the pass says nothing about it. Adding real Spanish cases makes the evidence cover current usage.

  • A ❌ Works when the suite covers today's usage; here a third of traffic is untested.
  • B ❌ Works for user behaviour once offline checks pass, but here it exposes Spanish speakers to an unchecked version.
  • C ✅ Add Spanish cases taken from real conversations to the suite, then rerun both versions.
  • D ❌ Works as a quick sense check, but a few impressions are anecdotes, not evidence.
Practice 9.3 · Choose one.

A logistics firm is moving its tracking assistant to a newer model version. The new version needs tool definitions in a new format, and the old format does not work with it. The team plans one release with both changes. A reviewer insists they must test one change at a time. What is the best approach?

  1. Test the new tool format alone on the current model, then add the model upgrade.
  2. Run the eval suite on the combined release against the current version, then stage it.
  3. Ship both changes together, since the provider recommends the newer model version.
  4. Test the model upgrade first on its own, then move to the new format in a later release.
Show answer

Answer: B. The deciding fact: the two changes cannot work apart, so they are one change. Test the pair as a unit on the eval suite, then roll out in stages.

  • A ❌ Works when each change stands alone; here the format exists only to serve the new model.
  • B ✅ Run the eval suite on the combined release against the current version, then stage it.
  • C ❌ Works, but a provider's recommendation is not evidence for your task; the eval must run first.
  • D ❌ Works for independent changes, but the new model cannot run with the old format, so this test is impossible.

Next trap: Cache it, don't cut it → · All traps