CCA-P start here · Trap gym · Trap 4

Check the inputs before the model

D4 · Evaluation, Testing & OptimizationGuide lines 4.4, 3.5

At a glance: When answers go wrong, look at what went into the model (the retrieved passages and the data) before blaming or replacing the model.

A RAG system (retrieval-augmented generation) finds passages in an index and gives them to Claude with the question. When an answer is wrong, there are three suspects in a fixed order: the inputs (did the right passage reach the model?), the prompt (was it there, but ignored or misused?) and only then the model (was everything right, and the task still too hard?).

The quickest way to tell them apart is to read the trace of a failing request: the record of what was retrieved and what was sent. If the needed fact is missing from the context, the cause is upstream: indexing, chunking, metadata or stale data. If the fact is present but the answer contradicts it, the prompt needs to ground the answer, for example by asking for supporting quotes and allowing “I don't know”. A model upgrade is the last step, confirmed by running the same failing cases on a more capable model.

The trap is that a model swap feels decisive. But a model cannot use a passage it never received, and a sudden failure usually follows something that changed: new data, a re-index, a new deployment.

Change one fact

A drug company's medical-information assistant answers doctors' questions from approved product leaflets. For one newly launched product, it states a dose that appears in no leaflet. Questions about other products are answered correctly. Nobody has looked at the traces yet. A team member proposes switching to the most capable model. What should the architect do first?

Words in the stem that signal this trap

Rule to remember: Read the traces first. Missing fact: fix retrieval or data. Fact present but ignored: fix the prompt. Only when both are right does a more capable model become the answer.

Practice

Practice 4.1 · Choose one.

A travel insurer's assistant answers questions about policy cover. Last month the team embedded newly added policies with a newer embedding model, but left the older policies and the query side unchanged. Since then, questions about the new policies return unrelated passages. What should the architect fix first?

  1. Add a stronger instruction telling the model to ignore unrelated passages.
  2. Embed queries and all documents with one model, then rebuild the index.
  3. Cut the new policies into smaller chunks with more overlap between them.
  4. Move the assistant to a more capable tier that handles policy wording better.
Show answer

Answer: B. The deciding fact: new documents and queries were embedded with different models, so their vectors no longer match. The inputs are wrong before the model sees anything.

  • A ❌ Works when the right passage arrives but is ignored; here it never arrives.
  • B ✅ Embed queries and all documents with one model, then rebuild the index.
  • C ❌ Works when ideas are split across chunks, but mismatched embeddings would still fail.
  • D ❌ Works when reasoning is the gap, but no model can use passages that were never retrieved.
Practice 4.2 · Choose one.

An energy trader's assistant explains why overnight positions breached risk limits, using the full risk report and limit rules, both sent whole. Traces confirm both are complete and correct, and the prompt was rewritten twice with worked examples. It explains single breaches well but gets cases wrong where three limits interact. What should the architect do next?

  1. Send just the rule that was breached, so the model has less text to read.
  2. Lower the temperature so that explanations are consistent between runs.
  3. Ask the model for a confidence score and hide low-confidence explanations.
  4. Run the failing cases on the most capable model with the same prompt and inputs.
Show answer

Answer: D. The deciding fact: inputs are confirmed correct and the prompt has been fixed twice. Upstream causes are ruled out, so test for a model mismatch on the same cases.

  • A ❌ Works for diluted attention, but these failures need several rules together.
  • B ❌ Works for consistency, but consistent output is not correct output.
  • C ❌ Works as one signal, but self-reported confidence is weak and fixes nothing.
  • D ✅ Run the failing cases on the most capable model with the same prompt and inputs.
Practice 4.3 · Choose two.

A hotel group's assistant answers guests' questions about each hotel's facilities. Guests at three recently opened hotels get answers describing a sister hotel's spa and pool. Answers for every other hotel are correct, and nothing else has changed. Which two checks should the architect make first?

  1. Whether the three new hotels' documents were indexed at all.
  2. Whether a more capable model answers the same questions better.
  3. Whether a lower temperature makes the answers more consistent.
  4. Whether a stricter system prompt reduces the wrong answers.
  5. Whether chunks carry a hotel tag and retrieval filters on it.
Show answer

Answer: A and E. The deciding fact: only the newly added hotels fail. That points to inputs: missing documents, or chunks that do not say which hotel they belong to.

  • A ✅ Whether the three new hotels' documents were indexed at all.
  • B ❌ Works only once inputs are ruled out; here they have not been checked.
  • C ❌ Works for consistency, but the answers are consistently about the wrong hotel.
  • D ❌ Works when the right passage arrives and is misused, which has not been checked.
  • E ✅ Whether chunks carry a hotel tag and retrieval filters on it.

Next trap: The least autonomy that works → · All traps