CCA-P start here · Trap gym · Trap 16

Measure fairness, don't prompt it

D5 · Governance, Safety & RiskGuide lines 5.5

At a glance: An instruction telling Claude to be fair proves nothing: measure outcomes across groups on representative data, look for proxies, tell people when AI is involved, and keep a named person accountable.

When leaders worry that an AI system treats some groups unfairly, the quickest-looking fix is a line in the system prompt: “treat everyone fairly”. It changes nothing you can show. Fairness is measured: run the system on a representative test set and compare outcomes (approval rates, scores, priorities) across groups. You cannot fix what you have not measured.

Measurement often finds bias that survived the obvious fix. Removing a protected attribute, such as ethnicity or gender, does not help if other fields stand in for it. These proxies include postcode, school, career gaps or names. Historical decisions used as examples can also carry past bias forward. The fix is to find the proxy, mask or replace it, and measure again.

Fairness also covers transparency and accountability. People should know when AI is involved, see the reasons, and be able to reach a person. For decisions about someone's job, credit or care, a named person makes and owns the decision; the AI supports. And the effort should match the stakes: a low-stakes internal tool needs light disclosure, not a group-outcome study.

Change one fact

A bank uses Claude to recommend which credit-card applications go to fast-track approval; the rest get a full manual review. Leadership worries it may be unfair to some groups. A product manager proposes adding “treat all applicants fairly regardless of background” to the system prompt. Nobody has yet compared outcomes across groups. What should the architect do first?

Words in the stem that signal this trap

Rule to remember: Fairness is shown by measurement, not promised in a prompt. Measure outcomes across groups on representative data, hunt proxies, disclose AI use, and keep a named person accountable for decisions about people.

Practice

Practice 16.1 · Choose one.

An insurer's assistant suggests home-insurance premiums for agents to review. The team reports no age bias, based on a test set of 2,000 past quotes. Fewer than 30 of those quotes were for customers over 65, though they make up a fifth of the insurer's customers. What should the architect do?

  1. Rebuild the test set so customers over 65 appear in proportion, then re-measure.
  2. Accept the result, since 2,000 quotes is a large enough test set for the whole book.
  3. Add an instruction telling the assistant to ignore the customer's age.
  4. Have an agent review every quote for a customer over 65 from now on.
Show answer

Answer: A. The deciding fact: the over-65 group is barely in the test set, so the “no bias” result says nothing about them. A representative set makes the measurement mean something.

  • A ✅ Rebuild the test set so customers over 65 appear in proportion, then re-measure.
  • B ❌ Works when every group is well represented; here 30 quotes cannot show a gap for a fifth of customers.
  • C ❌ Works as a hope, but it is untestable, and nobody has measured whether there is a problem.
  • D ❌ Works as a stopgap, but it treats one group differently without evidence that it is needed.
Practice 16.2 · Choose one.

An engineering firm's assistant drafts internal meeting notes from transcripts, and attendees edit them before sharing. A new ethics policy says every AI system needs a “fairness review”. The notes make no decisions about anyone. The architect must propose a review in proportion to the risk. Which fits best?

  1. A full study of the notes' outcomes across attendees' gender and ethnicity.
  2. A named manager who approves every set of notes before attendees see them.
  3. An instruction in the prompt telling the assistant to treat all speakers equally.
  4. A label saying the notes are AI-drafted, plus the attendees' existing edit step.
Show answer

Answer: D. The deciding fact: low-stakes notes that decide nothing about people, already edited by attendees. Light disclosure is proportionate; here the lighter option wins.

  • A ❌ Works for decisions about people, but these notes decide nothing, so the study measures little.
  • B ❌ Works for consequential decisions, but it slows a low-stakes tool attendees already edit.
  • C ❌ Works as a nudge, but it is untestable and adds nothing the edit step does not.
  • D ✅ A label saying the notes are AI-drafted, plus the attendees' existing edit step.
Practice 16.3 · Choose one.

A manufacturer will use an assistant to score staff for a redundancy programme against agreed criteria. HR proposes making the assistant's ranking final, arguing it is less biased than managers. Its scores have been measured as even across groups on a representative set. What should the architect recommend?

  1. Make the ranking final, since the measurement shows it is even across groups.
  2. Keep the ranking as advice; a named manager decides and records the reasons.
  3. Add an instruction that the assistant must weigh every employee's case fairly.
  4. Have a second model re-score every employee and use the average of the two.
Show answer

Answer: B. The deciding fact: the decision affects people's jobs. Even with fair measured scores, a named person must make and own it; the assistant supports.

  • A ❌ Works for low-stakes routing, but a decision about someone's job needs a person accountable.
  • B ✅ Keep the ranking as advice; a named manager decides and records the reasons.
  • C ❌ Works as a hope, but the scores are already measured, and it leaves nobody accountable.
  • D ❌ Works to reduce one model's noise, but two models still leave nobody accountable.

Next trap: Remove it, don't watch it → · All traps