Home › Hard problems › Inside the model

Inside the model: the core of LLMs

Hallucination: the model doesn't know what it doesn't know

Solo: strongOne person can run a real test of this in about 8 weeks.

In which year didthis tax rule change?"In 2014. Definitely."confidentlywrongTHE MODEL'S DASHBOARDsounds sure?"I'm guessing": no such light
Training rewards a confident answer, and the model has no reliable inner light that says "I am guessing".

Why it is still open

Training rewards a confident answer, and the model has no reliable internal signal for "I am guessing".

An experiment one person can run in 8 weeks

Take one domain (tax rules, drug doses). Build 500 questions with known answers. Measure how often each open model is confidently wrong, and test whether a cheap signal (asking 5 times and checking agreement, or reading internal activations) predicts those errors.

Verdict: Solo: strong

A domain test set with numbers is publishable and reusable.

The solo test behind the verdict

  1. Can it be measured? There is a number that shows you did better, such as an error rate, a catch rate or a cost per answer.
  2. Is it cheap to try? It runs on a laptop, free open models and open data, or under about ₹50,000 of rented GPU time.
  3. Is there a public yardstick? A benchmark or dataset others use, so your result can be compared and believed.
  4. Would a small win get noticed? A lab, a funder or a customer would care about a 10% improvement, not only a full solution.

Four yeses means Solo: strong. Two or three means Solo: possible. Fewer means Follow, don't build.

NextReliable multi-step reasoning

All 79 hard problems · Working on this, or know of a result we missed? basant@cleandataengineering.ai