CONCEPT Chunk size, overlap, rank, dedup
The retrieval half of a RAG pipeline (4.3) hinges on how you cut the corpus and which pieces you keep. Four knobs:
| Knob | What it controls | The tension |
|---|---|---|
| Chunk size | Granularity of each embedded/retrievable unit | Too small → context lost, claims un-embeddable; too big → noisy, dilutes the match |
| Overlap | Shared tokens between adjacent chunks | Prevents a fact from being split across a boundary; costs storage & near-duplicate hits |
| Ranking | Order candidates by relevance to the query | Embedding similarity is fast but coarse — a reranker (e.g. Voyage rerank-2.5) reorders the top candidates more precisely |
| Dedup | Drop near-identical chunks before assembly | Overlap and repeated boilerplate produce redundant hits that waste window budget |
CONCEPT Assemble so the best material is best-placed
Retrieval decides what enters the window; assembly decides where. Reuse the cluster's rules:
- Chunks near the top, query at the end — the query-at-the-end lift is up to 30% (4.1); it also keeps the most relevant chunks off the weakly-attended middle (4.2).
- Wrap each chunk in
<document>tags with a<source>so the model can attribute and, if you enable Citations, cite it (4.4). - Rank into position — put the highest-ranked chunk where attention is strongest; don't scatter the best evidence into the middle.
- Quote-first instruction last — ask Claude to extract the relevant passages before answering.
cache_control breakpoint, or you re-pay the write each call.DIAGRAM From corpus to a deliberately-built prompt
tagged w/ <source>, query at end,
quote-first instruction"] AS --> ANS["Grounded, well-attended answer"] classDef ok fill:#d4edda,stroke:#28a745,color:#155724; classDef gate fill:#ffe8cc,stroke:#ff9f40,color:#9a4a0a; classDef bad fill:#f8d7da,stroke:#dc3545,color:#721c24; class AS ok; class ANS ok;
CODE Rank, dedup, and assemble the final prompt
Rerank the retrieved candidates, drop near-duplicates, then place the survivors highest-first with the query last.
# candidates: [(chunk, embed_score), …] from the vector store ranked = vo.rerank(query, [c for c, _ in candidates], model="rerank-2.5").results # precise reorder seen, kept = set(), [] for r in ranked: # dedup near-identical key = r.document[:120] if key not in seen: seen.add(key); kept.append(r.document) # Assemble: highest-ranked chunk on TOP, each tagged, query at the END. blocks = [ {"type": "document", # citable per-chunk "source": {"type": "text", "media_type": "text/plain", "data": ch}, "title": f"chunk-{i}", "citations": {"enabled": True}} for i, ch in enumerate(kept) ] blocks.append({"type": "text", "text": f"Quote the passages relevant to: {query}\nThen answer."}) resp = client.messages.create(model="claude-opus-4-8", max_tokens=1024, messages=[{"role": "user", "content": blocks}])
TRADE-OFFS Small chunks vs. large chunks
✅ Smaller chunks + overlap + rerank
- Sharper embedding matches, less noise per hit
- Overlap keeps facts from splitting at boundaries
- Reranking recovers precision embeddings miss
⚠️ Large chunks, no rerank/dedup
- Fewer pieces to manage
- But each hit carries more irrelevant text into the window
- Duplicates and boilerplate waste budget and dilute attention
EXAM TRAPS
PRACTICE
Q1 · Why use chunk overlap?
So a fact that straddles a boundary isn't split across two chunks and lost to retrieval. Overlap shares tokens between adjacent chunks; the cost is extra storage and more near-duplicate hits — which is why you then dedup before assembly.
Q2 · You've retrieved 20 chunks. How should they enter the prompt?
Rerank them, dedup near-duplicates, then place the highest-ranked chunks on top (each wrapped in
<document>/<source> tags), with the query at the end and a quote-first
instruction. Assembly reuses the 4.1/4.2 placement rules.
Q3 · How do chunking choices interact with the Citations feature?
Put each retrieved chunk into its own plain-text or custom content document with citations
enabled — custom content is used as-is with no further chunking, so citations point at exactly the chunk you
retrieved. Remember the all-or-none rule and that citations can't be combined with
output_config.format.