D5 · Context & Reliability › Cluster 4 · Long Context & Retrieval

4.5 · Chunking & Context Assembly

Build the prompt deliberately

CONCEPT Chunk size, overlap, rank, dedup

The retrieval half of a RAG pipeline (4.3) hinges on how you cut the corpus and which pieces you keep. Four knobs:

KnobWhat it controlsThe tension
Chunk sizeGranularity of each embedded/retrievable unitToo small → context lost, claims un-embeddable; too big → noisy, dilutes the match
OverlapShared tokens between adjacent chunksPrevents a fact from being split across a boundary; costs storage & near-duplicate hits
RankingOrder candidates by relevance to the queryEmbedding similarity is fast but coarse — a reranker (e.g. Voyage rerank-2.5) reorders the top candidates more precisely
DedupDrop near-identical chunks before assemblyOverlap and repeated boilerplate produce redundant hits that waste window budget
🔑 Anthropic has no embedding model, so chunk sizing is really about the retriever you choose (e.g. Voyage). But the assembly — how the surviving chunks are placed in the prompt — is pure Anthropic long-context practice.

CONCEPT Assemble so the best material is best-placed

Retrieval decides what enters the window; assembly decides where. Reuse the cluster's rules:

For citable RAG, put each retrieved chunk into its own custom content document (no further chunking) or a plain-text document — then citations point back at exactly the chunk you retrieved.
Assembly is byte-order-sensitive for caching: keep the stable, ranked chunk prefix identical across turns and put the volatile query after the cache_control breakpoint, or you re-pay the write each call.

DIAGRAM From corpus to a deliberately-built prompt

flowchart TD C["Corpus"] --> CH["Chunk (size + overlap)"] CH --> E["Embed + retrieve top-k"] E --> RK["Rank / rerank"] RK --> DD["Dedup near-duplicates"] DD --> AS["🟢 Assemble: ranked chunks on top,
tagged w/ <source>, query at end,
quote-first instruction"] AS --> ANS["Grounded, well-attended answer"] classDef ok fill:#d4edda,stroke:#28a745,color:#155724; classDef gate fill:#ffe8cc,stroke:#ff9f40,color:#9a4a0a; classDef bad fill:#f8d7da,stroke:#dc3545,color:#721c24; class AS ok; class ANS ok;

CODE Rank, dedup, and assemble the final prompt

Rerank the retrieved candidates, drop near-duplicates, then place the survivors highest-first with the query last.

# candidates: [(chunk, embed_score), …] from the vector store
ranked = vo.rerank(query, [c for c, _ in candidates],
                  model="rerank-2.5").results   # precise reorder

seen, kept = set(), []
for r in ranked:                                  # dedup near-identical
    key = r.document[:120]
    if key not in seen:
        seen.add(key); kept.append(r.document)

# Assemble: highest-ranked chunk on TOP, each tagged, query at the END.
blocks = [
    {"type": "document",                          # citable per-chunk
     "source": {"type": "text", "media_type": "text/plain", "data": ch},
     "title": f"chunk-{i}", "citations": {"enabled": True}}
    for i, ch in enumerate(kept)
]
blocks.append({"type": "text",
               "text": f"Quote the passages relevant to: {query}\nThen answer."})

resp = client.messages.create(model="claude-opus-4-8",
    max_tokens=1024, messages=[{"role": "user", "content": blocks}])

TRADE-OFFS Small chunks vs. large chunks

✅ Smaller chunks + overlap + rerank

  • Sharper embedding matches, less noise per hit
  • Overlap keeps facts from splitting at boundaries
  • Reranking recovers precision embeddings miss

⚠️ Large chunks, no rerank/dedup

  • Fewer pieces to manage
  • But each hit carries more irrelevant text into the window
  • Duplicates and boilerplate waste budget and dilute attention

EXAM TRAPS

Trap: "Retrieval quality is the whole game." → Retrieval decides what enters the window; assembly decides where. Ranked chunks on top + query at the end still applies to a RAG prompt.
Trap: Skipping dedup after overlapping chunking. → Overlap and boilerplate produce near-duplicate hits that burn window budget and dilute attention — dedup before assembly.
Trap: Scattering the top-ranked chunk into the middle. → Put the strongest evidence at the high-attention edges (top/near-query), not buried mid-prompt (4.2).
Trap: Reordering ranked chunks every turn under a cache breakpoint. → A changed prefix invalidates the cache. Keep the ranked chunk block stable; put the varying query after the breakpoint.

PRACTICE

Q1 · Why use chunk overlap?

So a fact that straddles a boundary isn't split across two chunks and lost to retrieval. Overlap shares tokens between adjacent chunks; the cost is extra storage and more near-duplicate hits — which is why you then dedup before assembly.

Q2 · You've retrieved 20 chunks. How should they enter the prompt?

Rerank them, dedup near-duplicates, then place the highest-ranked chunks on top (each wrapped in <document>/<source> tags), with the query at the end and a quote-first instruction. Assembly reuses the 4.1/4.2 placement rules.

Q3 · How do chunking choices interact with the Citations feature?

Put each retrieved chunk into its own plain-text or custom content document with citations enabled — custom content is used as-is with no further chunking, so citations point at exactly the chunk you retrieved. Remember the all-or-none rule and that citations can't be combined with output_config.format.