D5 · Context & Reliability › Cluster 4 · Long Context & Retrieval

4.3 · RAG vs Long Context

Retrieve or stuff the window?

CONCEPT Two ways to get the right text in front of the model

You have a large corpus and a question. Two strategies:

The exam framing to internalize: a huge window does not remove the need for good retrieval. Every token you stuff is billed as input on every request, adds latency, and lands in a window where the middle is weakly attended (4.2). Retrieval trades a little recall risk for large cost, latency, and precision wins.

🔑 Anthropic does not offer its own embedding model. For the retrieval half you use a third-party provider — the docs recommend Voyage AI (e.g. voyage-4), with rerankers like rerank-2.5 to reorder candidates before they enter the prompt.

CONCEPT Choosing between them (and combining them)

DimensionRAG (retrieve chunks)Long context (stuff window)
Input cost / requestLow — only top-k chunks billedHigh — whole corpus billed every call
LatencyLower (small prompt)Higher (huge prompt to process)
Recall riskRetriever can miss a relevant chunkEverything is present (but middle under-attended)
SetupEmbeddings + vector store + rerankerJust build the prompt
Best whenCorpus ≫ window, cost/latency-sensitive, repeated queriesCorpus fits, cross-document reasoning, one-off deep read
They compose: retrieve a generous candidate set into a still-manageable window, then apply 4.1/4.2 placement (docs on top, query at the end, quote-first). RAG controls how much enters the window; long-context best practices control how well it's used.
Don't reach for the 1M window just because it exists. Stuffing 900K tokens you don't need pays full input cost per request and buries the answer in a weakly-attended middle.

DIAGRAM Routing the corpus into the prompt

flowchart TD Q["Query + large corpus"] --> D{"Corpus fits the window
AND cost is fine?"} D -->|"no — corpus ≫ window,
cost/latency matters"| RAG["🟢 RAG: embed (Voyage) →
top-k + rerank → small prompt"] D -->|"yes — cross-doc reasoning,
one-off deep read"| LC["🔵 Long context:
stuff the window"] RAG --> P["Assemble prompt
(4.1/4.2 placement)"] LC --> P P --> A["Grounded answer"] classDef ok fill:#d4edda,stroke:#28a745,color:#155724; classDef gate fill:#ffe8cc,stroke:#ff9f40,color:#9a4a0a; classDef bad fill:#f8d7da,stroke:#dc3545,color:#721c24; class RAG ok; class LC gate; class A ok;

CODE RAG: retrieve with Voyage, answer with Claude

Anthropic has no embedding endpoint, so the retrieval step uses Voyage. Note input_type differs for documents vs. the query — that's a Voyage requirement, not optional.

import voyageai, anthropic, numpy as np
vo, client = voyageai.Client(), anthropic.Anthropic()

# 1. Embed corpus + query (Anthropic offers no embedding model → Voyage).
doc_vecs = vo.embed(chunks, model="voyage-4", input_type="document").embeddings
q_vec    = vo.embed([query], model="voyage-4", input_type="query").embeddings[0]

# 2. Nearest neighbours → keep only the top-k relevant chunks.
scores = np.dot(doc_vecs, q_vec)              # cosine == dot (unit-normed)
top = [chunks[i] for i in np.argsort(scores)[-8:][::-1]]

# 3. Small prompt: retrieved chunks on top, query at the end (4.1).
resp = client.messages.create(
    model="claude-opus-4-8", max_tokens=1024,
    messages=[{"role": "user", "content":
        wrap_documents(top) + f"\n\nAnswer using only the passages above: {query}"}],
)

TRADE-OFFS Retrieve vs. stuff

✅ RAG

  • Cheap: only relevant chunks billed each call
  • Fast: small prompt, small window
  • Precise: irrelevant text never enters context
  • Scales past any window size

⚠️ Long context (stuff)

  • No retriever to miss a chunk (full recall present)
  • Great for cross-document synthesis
  • But: full input cost + latency every request
  • And: middle content weakly attended (4.2)

EXAM TRAPS

Trap: "The 1M window makes RAG obsolete." → False. Huge windows don't remove the need for good retrieval — stuffing pays full input cost per request and buries answers in the weakly-attended middle.
Trap: "Use Anthropic's embedding API for retrieval." → Anthropic has no embedding model. Use a third-party provider such as Voyage AI.
Trap: Ignoring per-request cost. → Stuffed context is billed on every call; RAG bills only the retrieved chunks. Prompt caching softens the stuffed case but doesn't beat retrieving less.
Trap: Embedding docs and the query with the same input_type. → Voyage wants input_type="document" for the corpus and "query" for the question — it improves retrieval quality.

PRACTICE

Q1 · When should you retrieve instead of stuffing the window?

When the corpus is far larger than the window, or when cost/latency matter, or the same corpus is queried repeatedly. Retrieval keeps the prompt small and precise. Stuff the window when the corpus fits and you need cross-document reasoning over the whole thing in one pass.

Q2 · Does Anthropic provide an embedding model for the RAG step?

No — Anthropic does not offer its own embedding model. Use a third-party provider; the docs recommend Voyage AI (e.g. voyage-4), optionally with a reranker like rerank-2.5.

Q3 · Why isn't a bigger context window a substitute for retrieval?

Because every stuffed token is billed per request and adds latency, and the extra text lands in a window whose middle is weakly attended. A bigger window raises the ceiling on what you can include; it does not make irrelevant text free or well-attended. Good retrieval still wins on cost, latency, and precision.