CONCEPT Examples steer — including in the wrong direction
Examples are the most reliable steer (3.1), which is exactly why bad examples steer wrongly. Because Claude pattern-matches whatever the examples share, any accidental trait, inconsistency, or specific content in your examples becomes a signal it follows. Three failure modes recur:
- Over-fitting (too few / too narrow) — with only a couple of similar examples, Claude latches onto their surface form (length, phrasing, an incidental shared category) instead of the underlying rule, and reproduces that form on inputs where it doesn't fit.
- Inconsistent formatting confuses — if examples use different tags, shapes, or output conventions, Claude gets a mixed signal about the intended format and its output becomes erratic.
- Content leakage — specific literal content from an example (a name, a number, a placeholder value, a canned phrase) can leak into the model's outputs for the real input, which should have had none of it.
| Pitfall | Symptom | Root cause | Fix |
|---|---|---|---|
| Over-fit | Outputs mimic example length/phrasing on all inputs | Too few / too similar examples | More examples, more diversity (3.2) |
| Inconsistent format | Erratic, unpredictable output shape | Examples don't match each other | Identical structure across all (3.3) |
| Leakage | An example's literal name/number appears in real output | Distinctive shared content | Vary/genericize the specifics |
DIAGRAM How a good steer goes bad
what they share"] M --> Q{"What do they share?"} Q -->|"the real rule + varied specifics"| GOOD["🟢 Generalizes correctly"] Q -->|"an incidental trait / narrow form"| OF["🟠 Over-fit to surface form"] Q -->|"a specific literal value"| LK["🔴 Content leaks into output"] Q -->|"conflicting formats"| CF["🟠 Erratic output shape"] classDef ok fill:#d4edda,stroke:#28a745,color:#155724; classDef gate fill:#ffe8cc,stroke:#ff9f40,color:#9a4a0a; classDef bad fill:#f8d7da,stroke:#dc3545,color:#721c24; class GOOD ok; class OF gate; class CF gate; class LK bad;
CODE Leakage & over-fit, then the fix
The bad set shares a distinctive literal ("Acme Corp") and a single narrow form, so Claude both over-fits the shape and leaks the name. The good set varies the specifics and broadens the range.
# BAD — one narrow shape + a distinctive literal repeated across examples bad = """<examples> <example><in>invoice from Acme Corp for $500</in><out>Acme Corp owes $500</out></example> <example><in>invoice from Acme Corp for $750</in><out>Acme Corp owes $750</out></example> </examples> <in>invoice from Zenith Ltd for $300</in>""" # Risk: over-fits the single phrasing; "Acme Corp" can LEAK into the Zenith output. # GOOD — varied vendors/amounts/phrasing; consistent tags; nothing distinctive to copy good = """<examples> <example><in>invoice from Acme Corp for $500</in><out>Acme Corp owes $500</out></example> <example><in>$750 billed by Zenith Ltd</in><out>Zenith Ltd owes $750</out></example> <example><in>Orbit Inc invoice, amount $1,200</in><out>Orbit Inc owes $1,200</out></example> </examples> <in>invoice from Vertex Co for $300</in>""" message = client.messages.create( model="claude-opus-4-8", max_tokens=16000, system="Extract who owes how much, following the examples' format.", messages=[{"role": "user", "content": good}], ) # → "Vertex Co owes $300" — rule generalized, no leaked literal, stable shape.
TRADE-OFFS The tension examples create
✅ Strong steer (the upside)
- Locks format, tone, and edge-case handling
- Improves accuracy & consistency (3.1)
⚠️ Strong steer (the risk)
- Too few / narrow → over-fits surface form
- Distinctive shared content → leaks into outputs
- Inconsistent formatting → erratic results
- Every example is also token cost
EXAM TRAPS
PRACTICE
Q1 · What is example over-fitting and what causes it?
Claude reproducing the surface form of the examples (length, phrasing, an incidental shared trait) instead of the underlying rule. It's caused by too few or too similar examples; the fix is more examples with more diversity so no accidental trait dominates.
Q2 · What is content leakage, and how do you prevent it?
Leakage is when specific literal content from an example — a name, number, or canned phrase — appears in the output for a real, unrelated input. Prevent it by making example specifics generic or varied across the set so no single value looks like it should be copied.
Q3 · Why does inconsistent formatting across examples hurt?
Consistent formatting is the signal Claude generalizes from. If examples disagree on tags, shape, or output convention, Claude gets a mixed signal and its output shape becomes erratic. Keep every example structurally identical.