CCA-F start here · Trap gym · Explicit criteria

Explicit criteria

A review bot flags too much, and developers stop reading it. The tempting answers ask Claude to be more careful: “be conservative”, “only report high-confidence findings”, a second pass to double-check. The exam's answer is to tell Claude exactly what counts as a finding, and what each severity level looks like in code.

D4 Prompt Engineering & Structured OutputExam guide 4.1About 15 minutes

At a glance

To cut false positives, give categorical criteria such as “flag a comment only when it contradicts the code”. Telling Claude to “be conservative” or to report “only high-confidence” findings doesn't raise precision.

Start here: the guard at the door

An office hires a new security guard for the front door.

“Be careful who you let in.” The guard stops the courier, the cleaner and a visitor in a suit, and waves through someone who looked confident. Telling them to be more careful just means more people stopped, for the same fuzzy reasons.
“Let in anyone with a blue badge. Stop anyone without one.” Now the guard knows exactly what counts. Add “visitors sign the book; contractors call facilities”, and every case has a clear action.

A review bot is the guard. A false positive is a finding that isn't a real problem, like a typo flagged as an “inaccurate comment”. Precision is the share of posted findings that are real. A mood word (“careful”, “conservative”, “confident”) doesn't tell Claude what counts, so the same kinds of noise keep coming. Explicit criteria (also called categorical criteria) do: they name what to flag, what to skip, and what each severity level looks like.

The noise also spreads. Developers don't read a review one category at a time. When half the comments are about typos, they skim the whole review, including the security finding that mattered.

Review prompt Doc-comment rule the wording you choose Security rule (already explicit) Performance rule (already explicit) Example findings optional: show layout and levels Pull request 40 spots to judge Claude writes findings Second pass optional re-check Posted findings inline PR comments Developer one trust for all Noise from one category makes the developer skim every category.
The one question to ask: does the fix tell Claude exactly what counts, or only how cautious to feel?

Words in the question that tell you which side you're on

Points to explicit criteria

  • “too many false positives”
  • “developers ignore the findings”, “skip its output”
  • “a noisy category”, “flags things that are fine”
  • a vague rule: “check comments are accurate”, “flag anything questionable”

Words in tempting wrong options

  • “be conservative”
  • “only high-confidence”
  • “be more careful”, “use your judgement”
  • “switch the category off”, “a second instance confirms each finding”

The whole idea on six comments

Below are six code comments from the same pull request, each judged by three versions of the review rule. The team's real question is only this: does the comment say something the code doesn't do? Two of the six do. The other four are fine.

Comment in the codeShould it be flagged?Rule ARule BRule C
// Returns null when not found, but the code throws an errorYes, the comment is falseFlags itFlags itFlags it
// Thread-safe, but the code uses a plain map with no lockYes, the comment is falseFlags itMisses it: the bug is subtle, so Claude isn't “highly confident”Flags it
// Recieve the confirmation (a spelling slip)No, it's still trueFlags it, noiseFlags it: Claude is very sure it's a typoLeaves it
A public function with no comment at allNo, nothing to contradictFlags it, noiseFlags it, noiseLeaves it
// Added for JIRA-1234 (2019)No, old but trueFlags it, noiseFlags it, noiseLeaves it
// Round down: partners round down tooNo, a good commentLeaves itLeaves itLeaves it
VerdictDrop. 2 real findings buried in 3 noisy onesDrop. Same noise, and it lost a real bugUse. 2 real findings, no noise. The exam's answer.

That's the whole trap. “Accurate” is a mood word: Claude has to guess what you mean, and it guesses that typos, missing comments and old tickets count. Adding “high-confidence” doesn't help, because Claude is most confident about the obvious noise, like a typo, and least confident about the subtle real bug. Only Rule C changes what counts, so only Rule C removes the noise.

Try it: review one pull request

The same idea at full size: the bot reviews 40 spots in a pull request for a travel-booking app. Watch the top row, “Doc comments”. That's the rule you change, and pink squares are noise. The security and performance rows have fixed, explicit rules. They are there to show one thing: the trust meter drops for all findings when one row is noisy. Click any square to see the code and what the bot posted.

Try the buttons in this order: “Check that comments are accurate”, then “+ Only report high-confidence findings”, then “Explicit criteria”. The last button, “+ severity by example”, adds a second, smaller point: once Claude flags the right things, concrete examples also make it label each finding with the right level (critical, major or minor). Squares with a dashed outline are real findings with the wrong level.

1. How the doc-comment rule is worded
2. Or keep the vague rule and work around it
The doc-comment part of the prompt
What the bot did with the 40 spots
flagged, rightlyfalse positivemissedleft alone, rightlyreal, wrong severity

Developer trust in the review /100

The behaviours and the trust figures are illustrative, chosen to show the shape. They are not measurements, and every run is the same. Trust here starts at 100 and drops 5 points per false positive and 4 per wrong severity. The exam guide's points are that explicit criteria beat vague instructions, that confidence wording doesn't improve precision like categorical criteria, and that a high false-positive category erodes trust in the accurate ones.

Things to try: start with “Check that comments are accurate” and watch the trust meter. Then + “Be conservative” and + “Only report high-confidence findings”: the pink barely thins, and click the yellow square to see what high-confidence lost. Then Explicit criteria, and click the dashed squares. Last, Explicit criteria + severity by example, and compare it with Drop the doc-comment category.

The four traps

TrapWhat the wrong option looks likeWhy it loses
Confidence wording only usually works“Be conservative”; “report only findings you are highly confident about”It says how cautious to be, not what counts. Claude is confident a typo is a typo, so the same kinds of noise stay.
Switch the category off delays the problemDrop the noisy category until its precision improvesThe noise goes, and so do the real bugs in that category. Explicit criteria keep the category and remove the noise.
A second pass overbuiltA second Claude call or instance double-checks each finding before it is postedIt re-checks against the same vague standard, for many more calls. Fix what gets flagged in the first place.
Examples for a vague rule different problemAdd a few example findings to the promptExamples mainly fix inconsistent format and borderline calls. When the flagging rule itself is vague, state the criteria first.

Practice: three exam-style questions

Answer first, then open “Think it through”.

Rule to remember

Tell Claude exactly what to flag; telling it to be careful doesn't cut false positives. Name what to flag and what to skip, and define each severity with a concrete code example.

Exam guide 4.1: “Explicit criteria beat vague instructions ("flag comments only when claimed behavior contradicts actual code" vs "check comments are accurate").” “"Be conservative" / "only high-confidence" don't improve precision like categorical criteria.” “High false-positive categories erode developer trust in accurate ones; define explicit severity criteria with concrete code examples.”

← All traps