Explicit criteria
A review bot flags too much, and developers stop reading it. The tempting answers ask Claude to be more careful: “be conservative”, “only report high-confidence findings”, a second pass to double-check. The exam's answer is to tell Claude exactly what counts as a finding, and what each severity level looks like in code.
D4 Prompt Engineering & Structured OutputExam guide 4.1About 15 minutes
At a glance
Start here: the guard at the door
An office hires a new security guard for the front door.
A review bot is the guard. A false positive is a finding that isn't a real problem, like a typo flagged as an “inaccurate comment”. Precision is the share of posted findings that are real. A mood word (“careful”, “conservative”, “confident”) doesn't tell Claude what counts, so the same kinds of noise keep coming. Explicit criteria (also called categorical criteria) do: they name what to flag, what to skip, and what each severity level looks like.
The noise also spreads. Developers don't read a review one category at a time. When half the comments are about typos, they skim the whole review, including the security finding that mattered.
- Only how cautious (“be conservative”, “only high-confidence”, “be more careful”, a second pass with the same rule): precision barely moves. Claude is confident about the wrong things.
- Exactly what counts: name what to flag and what to skip (“flag a comment only when its claimed behaviour contradicts the code”), and define each severity level with a concrete code example. This is the exam's answer.
- The category is noisy but catches real bugs: rewrite its criteria. Don't switch it off.
- The rule is already clear, but the format or the borderline calls vary: that is a different trap. Add a few examples (see Few-shot examples).
Words in the question that tell you which side you're on
Points to explicit criteria
- “too many false positives”
- “developers ignore the findings”, “skip its output”
- “a noisy category”, “flags things that are fine”
- a vague rule: “check comments are accurate”, “flag anything questionable”
Words in tempting wrong options
- “be conservative”
- “only high-confidence”
- “be more careful”, “use your judgement”
- “switch the category off”, “a second instance confirms each finding”
The whole idea on six comments
Below are six code comments from the same pull request, each judged by three versions of the review rule. The team's real question is only this: does the comment say something the code doesn't do? Two of the six do. The other four are fine.
- Rule A (vague): “Check that comments are accurate.”
- Rule B (vague, plus confidence wording): Rule A + “Only report findings you are highly confident about.”
- Rule C (explicit criteria): “Flag a comment only when what it claims contradicts what the code does. Don't flag spelling, missing comments, old-looking wording or TODOs.”
| Comment in the code | Should it be flagged? | Rule A | Rule B | Rule C |
|---|---|---|---|---|
// Returns null when not found, but the code throws an error | Yes, the comment is false | Flags it | Flags it | Flags it |
// Thread-safe, but the code uses a plain map with no lock | Yes, the comment is false | Flags it | Misses it: the bug is subtle, so Claude isn't “highly confident” | Flags it |
// Recieve the confirmation (a spelling slip) | No, it's still true | Flags it, noise | Flags it: Claude is very sure it's a typo | Leaves it |
| A public function with no comment at all | No, nothing to contradict | Flags it, noise | Flags it, noise | Leaves it |
// Added for JIRA-1234 (2019) | No, old but true | Flags it, noise | Flags it, noise | Leaves it |
// Round down: partners round down too | No, a good comment | Leaves it | Leaves it | Leaves it |
| Verdict | Drop. 2 real findings buried in 3 noisy ones | Drop. Same noise, and it lost a real bug | Use. 2 real findings, no noise. The exam's answer. |
That's the whole trap. “Accurate” is a mood word: Claude has to guess what you mean, and it guesses that typos, missing comments and old tickets count. Adding “high-confidence” doesn't help, because Claude is most confident about the obvious noise, like a typo, and least confident about the subtle real bug. Only Rule C changes what counts, so only Rule C removes the noise.
Try it: review one pull request
The same idea at full size: the bot reviews 40 spots in a pull request for a travel-booking app. Watch the top row, “Doc comments”. That's the rule you change, and pink squares are noise. The security and performance rows have fixed, explicit rules. They are there to show one thing: the trust meter drops for all findings when one row is noisy. Click any square to see the code and what the bot posted.
Try the buttons in this order: “Check that comments are accurate”, then “+ Only report high-confidence findings”, then “Explicit criteria”. The last button, “+ severity by example”, adds a second, smaller point: once Claude flags the right things, concrete examples also make it label each finding with the right level (critical, major or minor). Squares with a dashed outline are real findings with the wrong level.
The behaviours and the trust figures are illustrative, chosen to show the shape. They are not measurements, and every run is the same. Trust here starts at 100 and drops 5 points per false positive and 4 per wrong severity. The exam guide's points are that explicit criteria beat vague instructions, that confidence wording doesn't improve precision like categorical criteria, and that a high false-positive category erodes trust in the accurate ones.
Things to try: start with “Check that comments are accurate” and watch the trust meter. Then + “Be conservative” and + “Only report high-confidence findings”: the pink barely thins, and click the yellow square to see what high-confidence lost. Then Explicit criteria, and click the dashed squares. Last, Explicit criteria + severity by example, and compare it with Drop the doc-comment category.
The four traps
| Trap | What the wrong option looks like | Why it loses |
|---|---|---|
| Confidence wording only usually works | “Be conservative”; “report only findings you are highly confident about” | It says how cautious to be, not what counts. Claude is confident a typo is a typo, so the same kinds of noise stay. |
| Switch the category off delays the problem | Drop the noisy category until its precision improves | The noise goes, and so do the real bugs in that category. Explicit criteria keep the category and remove the noise. |
| A second pass overbuilt | A second Claude call or instance double-checks each finding before it is posted | It re-checks against the same vague standard, for many more calls. Fix what gets flagged in the first place. |
| Examples for a vague rule different problem | Add a few example findings to the prompt | Examples mainly fix inconsistent format and borderline calls. When the flagging rule itself is vague, state the criteria first. |
Practice: three exam-style questions
Answer first, then open “Think it through”.
Rule to remember
Exam guide 4.1: “Explicit criteria beat vague instructions ("flag comments only when claimed behavior contradicts actual code" vs "check comments are accurate").” “"Be conservative" / "only high-confidence" don't improve precision like categorical criteria.” “High false-positive categories erode developer trust in accurate ones; define explicit severity criteria with concrete code examples.”