Home › Hard problems › Inside the model

Inside the model: the core of LLMs

Reliable multi-step reasoning

Solo: strongOne person can run a real test of this in about 8 weeks.

startanswerstep 1step 2step 3step 4step 5step 6step 7step 8step 9step 1095%about 60%each bar: the chance every step so far was right
Small errors compound: 95% right per step is about 60% right over 10 steps. And the written reasoning may not match how the model really got there.

Why it is still open

Small errors compound: 95% right per step is about 60% right over 10 steps. Models also give reasoning that doesn't match how they reached the answer.

An experiment one person can run in 8 weeks

Measure how success drops with task length for 5 models on one task family (spreadsheet edits, multi-hop questions). Test one fix, such as checking each step with a second model.

Verdict: Solo: strong

The "error by length" curve is a known gap with few public measurements per domain.

The solo test behind the verdict

  1. Can it be measured? There is a number that shows you did better, such as an error rate, a catch rate or a cost per answer.
  2. Is it cheap to try? It runs on a laptop, free open models and open data, or under about ₹50,000 of rented GPU time.
  3. Is there a public yardstick? A benchmark or dataset others use, so your result can be compared and believed.
  4. Would a small win get noticed? A lab, a funder or a customer would care about a 10% improvement, not only a full solution.

Four yeses means Solo: strong. Two or three means Solo: possible. Fewer means Follow, don't build.

PreviousHallucination: the model doesn't know what it doesn't knowNextThinking cost: models overthink easy questions

All 79 hard problems · Working on this, or know of a result we missed? basant@cleandataengineering.ai