D4 · Prompt Engineering › Cluster 3 · Examples (Multishot)

3.5 · Example Pitfalls (Leakage, Over-fit)

When examples backfire

CONCEPT Examples steer — including in the wrong direction

Examples are the most reliable steer (3.1), which is exactly why bad examples steer wrongly. Because Claude pattern-matches whatever the examples share, any accidental trait, inconsistency, or specific content in your examples becomes a signal it follows. Three failure modes recur:

🔑 The fix for all three is the discipline from 3.2–3.3: use enough examples (3–5), keep them diverse (so no accidental trait dominates), keep their formatting consistent, and make each example's specifics generic or varied so no single value looks like it should be copied.
PitfallSymptomRoot causeFix
Over-fitOutputs mimic example length/phrasing on all inputsToo few / too similar examplesMore examples, more diversity (3.2)
Inconsistent formatErratic, unpredictable output shapeExamples don't match each otherIdentical structure across all (3.3)
LeakageAn example's literal name/number appears in real outputDistinctive shared contentVary/genericize the specifics

DIAGRAM How a good steer goes bad

flowchart TD EX["Examples (strong steer)"] --> M["Claude pattern-matches
what they share"] M --> Q{"What do they share?"} Q -->|"the real rule + varied specifics"| GOOD["🟢 Generalizes correctly"] Q -->|"an incidental trait / narrow form"| OF["🟠 Over-fit to surface form"] Q -->|"a specific literal value"| LK["🔴 Content leaks into output"] Q -->|"conflicting formats"| CF["🟠 Erratic output shape"] classDef ok fill:#d4edda,stroke:#28a745,color:#155724; classDef gate fill:#ffe8cc,stroke:#ff9f40,color:#9a4a0a; classDef bad fill:#f8d7da,stroke:#dc3545,color:#721c24; class GOOD ok; class OF gate; class CF gate; class LK bad;

CODE Leakage & over-fit, then the fix

The bad set shares a distinctive literal ("Acme Corp") and a single narrow form, so Claude both over-fits the shape and leaks the name. The good set varies the specifics and broadens the range.

# BAD — one narrow shape + a distinctive literal repeated across examples
bad = """<examples>
  <example><in>invoice from Acme Corp for $500</in><out>Acme Corp owes $500</out></example>
  <example><in>invoice from Acme Corp for $750</in><out>Acme Corp owes $750</out></example>
</examples>

<in>invoice from Zenith Ltd for $300</in>"""
# Risk: over-fits the single phrasing; "Acme Corp" can LEAK into the Zenith output.

# GOOD — varied vendors/amounts/phrasing; consistent tags; nothing distinctive to copy
good = """<examples>
  <example><in>invoice from Acme Corp for $500</in><out>Acme Corp owes $500</out></example>
  <example><in>$750 billed by Zenith Ltd</in><out>Zenith Ltd owes $750</out></example>
  <example><in>Orbit Inc invoice, amount $1,200</in><out>Orbit Inc owes $1,200</out></example>
</examples>

<in>invoice from Vertex Co for $300</in>"""

message = client.messages.create(
    model="claude-opus-4-8",
    max_tokens=16000,
    system="Extract who owes how much, following the examples' format.",
    messages=[{"role": "user", "content": good}],
)
# → "Vertex Co owes $300"  — rule generalized, no leaked literal, stable shape.

TRADE-OFFS The tension examples create

✅ Strong steer (the upside)

  • Locks format, tone, and edge-case handling
  • Improves accuracy & consistency (3.1)

⚠️ Strong steer (the risk)

  • Too few / narrow → over-fits surface form
  • Distinctive shared content → leaks into outputs
  • Inconsistent formatting → erratic results
  • Every example is also token cost
The steer's strength is the risk. Manage it with count, diversity, consistency, and generic specifics — the same levers from 3.2 and 3.3, applied defensively.

EXAM TRAPS

Trap: "More examples always help." → Too few/narrow over-fits, but redundant or inconsistent examples also hurt. It's diversity and consistency that matter, not raw count.
Trap: "Reusing a real record as an example is fine." → Its literal content (names, numbers, canned phrases) can leak into outputs for unrelated inputs. Genericize or vary the specifics.
Trap: "Format inconsistency is harmless as long as content is right." → Inconsistent example formatting confuses Claude and makes output shape erratic. Keep every example structurally identical.
Trap: "If examples backfire, the technique is unreliable." → Examples are the most reliable steer; backfires come from bad example design (too few, narrow, inconsistent, leaky), all of which are fixable.

PRACTICE

Q1 · What is example over-fitting and what causes it?

Claude reproducing the surface form of the examples (length, phrasing, an incidental shared trait) instead of the underlying rule. It's caused by too few or too similar examples; the fix is more examples with more diversity so no accidental trait dominates.

Q2 · What is content leakage, and how do you prevent it?

Leakage is when specific literal content from an example — a name, number, or canned phrase — appears in the output for a real, unrelated input. Prevent it by making example specifics generic or varied across the set so no single value looks like it should be copied.

Q3 · Why does inconsistent formatting across examples hurt?

Consistent formatting is the signal Claude generalizes from. If examples disagree on tags, shape, or output convention, Claude gets a mixed signal and its output shape becomes erratic. Keep every example structurally identical.