Match the search to the question
D3 · IntegrationGuide lines 3.6, 3.5
Vector search turns text into numbers that capture meaning, so “car won't start” can find “engine fails to turn over”. That is its strength and its weakness: an exact code such as RA-558201 has no meaning to match, so vector search returns things that look similar instead of the exact record. Keyword search (often BM25, a standard way of ranking documents by matching words) finds exact terms but misses paraphrases. Hybrid search runs both and merges the results.
Some questions are not search questions at all. “How many cars were out of service last month?” needs every row counted exactly, which is a job for a query tool over the database, not for the twenty most similar text chunks. And when results are the right kind of text from the wrong branch, region or year, the fix is metadata: tag each chunk with those fields and filter on them.
A reranker re-scores results that were already retrieved. It helps when the right result is found but ranked too low; it cannot help when the right result was never retrieved or was filtered wrongly.
Change one fact
A car-rental company's assistant answers branch staff's questions from one vector index built over rental agreements, fleet records and customer reviews. The failing questions ask for counts and totals, such as “How many cars at the airport branch were out of service last month?”, and get wrong numbers. Which retrieval change should the architect make?
Words in the stem that signal this trap
- exact codes, IDs, SKUs or citations in the query
- “how many”, “total”, “by region” over table data
- “right text but the wrong country, version or year”
- “the correct passage was in the top 50 but not the top 5” (a reranker)
- users describing needs in their own words (vector search fits)
Practice
A garden centre's assistant answers questions such as “What can I plant in a shady, damp corner?” from 3,000 plant-care guides. Customers rarely use plant names, and the guides describe growing conditions in many different words. A colleague proposes replacing vector search with keyword search to make results more exact. What should the architect do?
- Replace vector search with keyword search across all of the plant-care guides.
- Load the guides into a database and give the assistant a SQL query tool.
- Filter every search by plant name before the results are ranked.
- Keep vector search, because customers describe their needs in everyday words.
Show answer
Answer: D. The deciding fact: questions describe needs in varied wording, without names or codes. Matching meaning is what vector search is for.
- A ❌ Works for exact identifiers, but here it would miss guides that use different words.
- B ❌ Works for counts and totals, but these questions need descriptive text.
- C ❌ Works when names are known, but customers rarely give one.
- D ✅ Keep vector search, because customers describe their needs in everyday words.
An engineering firm's assistant searches 80,000 paragraphs of design standards. Logs show that for most poor answers, the correct paragraph was among the top 50 vector results but not in the top five passed to Claude. Queries are plain-language descriptions with no codes. What should the architect add?
- Keyword search alongside vector search, with the two result lists merged.
- A reranker that re-scores the top 50 results and passes on the best five.
- A larger embedding model, followed by a full rebuild of the index.
- A change that sends all 50 retrieved paragraphs to Claude on every query.
Show answer
Answer: B. The deciding fact: the right paragraph is retrieved but ranked too low. A reranker improves precision on results that are already there.
- A ❌ Works when queries contain exact terms, but these are plain descriptions.
- B ✅ A reranker that re-scores the top 50 results and passes on the best five.
- C ❌ Works sometimes, but the right result is already found; the ranking is the gap.
- D ❌ Works partly, but ten times the text adds cost and dilutes attention.
A mobile operator's assistant handles two kinds of question: “What does my contract say about roaming?” and “How much data did I use each day last week?”. Both are answered from one vector index built over contract text and usage rows. Usage answers are often wrong. Which two changes fit best?
- Embed the usage rows with a larger embedding model to improve the matches.
- Add a reranker over the usage rows that vector search returns.
- Keep vector search over contract text for the contract questions.
- Paste the customer's full usage history into the prompt for every question.
- Answer usage questions with a query tool over the usage table.
Show answer
Answer: C and E. The deciding fact: two question shapes. Contract wording suits vector search; daily usage figures need exact values from the table, so a query tool.
- A ❌ Works for meaning, but a better embedding still cannot compute daily totals.
- B ❌ Works when the right result ranks low, not when exact figures are needed.
- C ✅ Keep vector search over contract text for the contract questions.
- D ❌ Works for a short history, but it is wasteful on every question, including contract ones.
- E ✅ Answer usage questions with a query tool over the usage table.