CCA-P start here · Trap gym · Trap 15

Review where the risk is

D5 · Governance, Safety & RiskGuide lines 5.3

At a glance: Neither review everything nor review nothing: approval before irreversible or high-impact actions, automation plus random audits for low-risk volume, and full review only when every output carries real weight.

Human-in-the-loop means a person checks or approves some of what an AI system does. The two easy answers are both traps. Reviewing everything sounds safe, but at high volume reviewers start rubber-stamping (approving without really checking), so the control stops working while the cost stays. Reviewing nothing leaves the rare expensive mistake unguarded.

The fix is to place people by risk. Actions that cannot be undone or carry high value need a person's approval before they happen, because an audit afterwards cannot reverse the harm. Low-value, reversible, high-volume work runs automatically, and a random sample is audited to measure the true error rate. Routing rules, such as an amount threshold, are enforced in code, not left to the model's own sense of confidence, which can be high even when it is wrong.

Sometimes the strict answer is right: when every output carries legal or safety weight, every one is reviewed. And review works better when reviewers see the evidence next to the summary, and when their corrections are kept as data to improve the system.

Change one fact

A hospital supply agent reorders stock by placing purchase orders with approved suppliers, about 2,000 a week. Most are routine consumables under $500. A few each week are for equipment over $50,000, and an order cannot be cancelled once the supplier ships. Leadership wants as much automation as is safe. Which review design fits best?

Words in the stem that signal this trap

Rule to remember: Put people where an error is costly and cannot be undone, before the action. Automate the low-risk volume and audit a random sample. Route on rules in code, not the model's own confidence.

Practice

Practice 15.1 · Choose one.

An airline's maintenance assistant drafts the records that release an aircraft back into service after repairs. Aviation rules require a licensed engineer to certify each release, and a wrong record could let an unsafe aircraft fly. The airline wants to cut the engineers' paperwork time. Which review design fits best?

  1. Engineers audit a random sample of the records released, at the end of every week.
  2. Engineers review only the drafts that the assistant itself marks as uncertain.
  3. An engineer checks and certifies every draft, with the evidence shown beside it.
  4. Records for minor repairs release automatically; major ones go to an engineer.
Show answer

Answer: C. The deciding fact: every record carries safety and legal weight, and the rules require certification of each one. The saving comes from showing evidence, not from skipping review.

  • A ❌ Works for low-risk volume, but an unsafe aircraft could fly before the audit.
  • B ❌ Works poorly anywhere; the assistant can be confidently wrong, and rules require every record certified.
  • C ✅ An engineer checks and certifies every draft, with the evidence shown beside it.
  • D ❌ Works when small items are low risk, but every release here needs a licensed engineer.
Practice 15.2 · Choose one.

A bank's loan-modification assistant prepares cases that an officer must approve, because each decision changes a customer's repayments. Officers spend about twenty minutes per case searching the customer's documents for the figures behind the assistant's summary, and the backlog is growing. What should the architect change?

  1. Let the assistant approve cases it rates as high confidence, and send the rest to officers.
  2. Approve cases automatically and have officers audit a random sample each week.
  3. Add more officers to the team until the backlog falls to an acceptable level.
  4. Show each figure in the summary next to the passage of the document it came from.
Show answer

Answer: D. The deciding fact: every case still needs an officer, and the time goes into finding the evidence. Putting the evidence beside the summary makes each review faster and better.

  • A ❌ Works poorly; confidence is a weak signal, and these decisions need an officer's approval.
  • B ❌ Works for low-impact volume, but each decision changes a customer's repayments.
  • C ❌ Works to clear the queue, but it pays for the same slow search on every case.
  • D ✅ Show each figure in the summary next to the passage of the document it came from.
Practice 15.3 · Choose one.

An insurer's claims assistant drafts letters that handlers review before sending. Handlers correct about 8% of drafts in the review tool, but the corrections are discarded. Next quarter, leadership wants to stop reviewing the claim categories that are safe to automate. What should the architect put in place now?

  1. A monthly survey asking handlers how far they now trust the assistant's drafts.
  2. Keep each correction with its category, and measure error rates per category.
  3. Stop reviewing the claim category that draws the fewest customer complaints.
  4. A second model that checks every draft, so that handlers can be removed.
Show answer

Answer: B. The deciding fact: the corrections already exist and show where errors happen. Keeping them by category gives the evidence to decide where review can safely stop.

  • A ❌ Works to gauge morale, but trust is self-reported, not a measured error rate.
  • B ✅ Keep each correction with its category, and measure error rates per category.
  • C ❌ Works as a hint, but few complaints can mean errors go unnoticed, not that none happen.
  • D ❌ Works as an extra check, but it removes people before anyone knows the error rates.

Next trap: Measure fairness, don't prompt it → · All traps