Home › Hard problems › Data and computer systems

Data and computer systems

Retrieval that finds the right passage

Solo: strongOne person can run a real test of this in about 8 weeks.

A question"What was the Q3GST for plant B?"Top 5 fetched1. Q3 sales summary2. GST rules, general3. plant A budget4. Q2 GST letter5. B-team org chartSay, rank 9: the right onea table of numbersThe modelanswers from thewrong textAn answerwrong, butsounds suresearch slips on:tablesnumbersacronymsmixed languages
Most wrong RAG answers start with fetching the wrong text. The right passage, often a table or a number, never reaches the model.

Why it is still open

Most wrong RAG answers come from fetching the wrong text. Search quality drops on tables, numbers, acronyms and mixed languages.

An experiment one person can run in 8 weeks

Build 300 questions over messy enterprise-style documents and compare 4 retrieval setups on whether the right passage is in the top 5.

Verdict: Solo: strong

Measurable and directly useful.

The solo test behind the verdict

  1. Can it be measured? There is a number that shows you did better, such as an error rate, a catch rate or a cost per answer.
  2. Is it cheap to try? It runs on a laptop, free open models and open data, or under about ₹50,000 of rented GPU time.
  3. Is there a public yardstick? A benchmark or dataset others use, so your result can be compared and believed.
  4. Would a small win get noticed? A lab, a funder or a customer would care about a 10% improvement, not only a full solution.

Four yeses means Solo: strong. Two or three means Solo: possible. Fewer means Follow, don't build.

PreviousSame input, different LLM answerNextMatching records that refer to the same thing

All 79 hard problems · Working on this, or know of a result we missed? basant@cleandataengineering.ai