Home › Hard problems › Data and computer systems
Data and computer systems
Matching records that refer to the same thing
Solo: strongOne person can run a real test of this in about 8 weeks.
Why it is still open
The same customer, drug or company is spelled differently across systems ("entity resolution"), and it still needs people to check at scale.
An experiment one person can run in 8 weeks
On a public benchmark, compare a rules approach, an embedding approach and an LLM judge on accuracy and cost per million pairs.
Verdict: Solo: strong
Your home ground, and the cost numbers are new.
The solo test behind the verdict
- Can it be measured? There is a number that shows you did better, such as an error rate, a catch rate or a cost per answer.
- Is it cheap to try? It runs on a laptop, free open models and open data, or under about ₹50,000 of rented GPU time.
- Is there a public yardstick? A benchmark or dataset others use, so your result can be compared and believed.
- Would a small win get noticed? A lab, a funder or a customer would care about a 10% improvement, not only a full solution.
Four yeses means Solo: strong. Two or three means Solo: possible. Fewer means Follow, don't build.
All 79 hard problems · Working on this, or know of a result we missed? basant@cleandataengineering.ai