CONCEPT Two ways to get the right text in front of the model
You have a large corpus and a question. Two strategies:
- Retrieve (RAG): embed the corpus, find the few chunks most relevant to the query, and put only those into a small prompt.
- Long context: stuff the whole corpus (or a big slice) into a large window — Opus 4.8 offers a 1M-token context window — and let the model read all of it.
The exam framing to internalize: a huge window does not remove the need for good retrieval. Every token you stuff is billed as input on every request, adds latency, and lands in a window where the middle is weakly attended (4.2). Retrieval trades a little recall risk for large cost, latency, and precision wins.
voyage-4), with rerankers like
rerank-2.5 to reorder candidates before they enter the prompt.CONCEPT Choosing between them (and combining them)
| Dimension | RAG (retrieve chunks) | Long context (stuff window) |
|---|---|---|
| Input cost / request | Low — only top-k chunks billed | High — whole corpus billed every call |
| Latency | Lower (small prompt) | Higher (huge prompt to process) |
| Recall risk | Retriever can miss a relevant chunk | Everything is present (but middle under-attended) |
| Setup | Embeddings + vector store + reranker | Just build the prompt |
| Best when | Corpus ≫ window, cost/latency-sensitive, repeated queries | Corpus fits, cross-document reasoning, one-off deep read |
DIAGRAM Routing the corpus into the prompt
AND cost is fine?"} D -->|"no — corpus ≫ window,
cost/latency matters"| RAG["🟢 RAG: embed (Voyage) →
top-k + rerank → small prompt"] D -->|"yes — cross-doc reasoning,
one-off deep read"| LC["🔵 Long context:
stuff the window"] RAG --> P["Assemble prompt
(4.1/4.2 placement)"] LC --> P P --> A["Grounded answer"] classDef ok fill:#d4edda,stroke:#28a745,color:#155724; classDef gate fill:#ffe8cc,stroke:#ff9f40,color:#9a4a0a; classDef bad fill:#f8d7da,stroke:#dc3545,color:#721c24; class RAG ok; class LC gate; class A ok;
CODE RAG: retrieve with Voyage, answer with Claude
Anthropic has no embedding endpoint, so the retrieval step uses Voyage. Note input_type differs for
documents vs. the query — that's a Voyage requirement, not optional.
import voyageai, anthropic, numpy as np vo, client = voyageai.Client(), anthropic.Anthropic() # 1. Embed corpus + query (Anthropic offers no embedding model → Voyage). doc_vecs = vo.embed(chunks, model="voyage-4", input_type="document").embeddings q_vec = vo.embed([query], model="voyage-4", input_type="query").embeddings[0] # 2. Nearest neighbours → keep only the top-k relevant chunks. scores = np.dot(doc_vecs, q_vec) # cosine == dot (unit-normed) top = [chunks[i] for i in np.argsort(scores)[-8:][::-1]] # 3. Small prompt: retrieved chunks on top, query at the end (4.1). resp = client.messages.create( model="claude-opus-4-8", max_tokens=1024, messages=[{"role": "user", "content": wrap_documents(top) + f"\n\nAnswer using only the passages above: {query}"}], )
TRADE-OFFS Retrieve vs. stuff
✅ RAG
- Cheap: only relevant chunks billed each call
- Fast: small prompt, small window
- Precise: irrelevant text never enters context
- Scales past any window size
⚠️ Long context (stuff)
- No retriever to miss a chunk (full recall present)
- Great for cross-document synthesis
- But: full input cost + latency every request
- And: middle content weakly attended (4.2)
EXAM TRAPS
input_type. → Voyage wants
input_type="document" for the corpus and "query" for the question — it improves retrieval
quality.PRACTICE
Q1 · When should you retrieve instead of stuffing the window?
When the corpus is far larger than the window, or when cost/latency matter, or the same corpus is queried repeatedly. Retrieval keeps the prompt small and precise. Stuff the window when the corpus fits and you need cross-document reasoning over the whole thing in one pass.
Q2 · Does Anthropic provide an embedding model for the RAG step?
No — Anthropic does not offer its own embedding model. Use a third-party provider; the docs
recommend Voyage AI (e.g. voyage-4), optionally with a reranker like rerank-2.5.
Q3 · Why isn't a bigger context window a substitute for retrieval?
Because every stuffed token is billed per request and adds latency, and the extra text lands in a window whose middle is weakly attended. A bigger window raises the ceiling on what you can include; it does not make irrelevant text free or well-attended. Good retrieval still wins on cost, latency, and precision.