Home › Hard problems › Inside the model

Inside the model: the core of LLMs

Evaluation itself is broken

Solo: strongOne person can run a real test of this in about 8 weeks.

1. THE TEST LEAKS INTO TRAININGTESTquestions + answerspostedonlinetraining data"98%! (seen it)"2. EVERY MODEL HITS THE TOP WITHIN MONTHSmaxmodel Amodel Bmodel Cmodel Dso which one is best for my task?
Benchmarks leak into training data and saturate within months, so scores stop telling you which model is better for your task.

Why it is still open

Benchmarks leak into training data and saturate within months. Scores stop telling you which model is better for your task.

An experiment one person can run in 8 weeks

Write fresh questions after a model's cutoff date and compare its score on old against new questions in one area. The gap estimates contamination.

Verdict: Solo: strong

Cheap and needed by every buyer.

The solo test behind the verdict

  1. Can it be measured? There is a number that shows you did better, such as an error rate, a catch rate or a cost per answer.
  2. Is it cheap to try? It runs on a laptop, free open models and open data, or under about ₹50,000 of rented GPU time.
  3. Is there a public yardstick? A benchmark or dataset others use, so your result can be compared and believed.
  4. Would a small win get noticed? A lab, a funder or a customer would care about a 10% improvement, not only a full solution.

Four yeses means Solo: strong. Two or three means Solo: possible. Fewer means Follow, don't build.

PreviousRunning out of human text; synthetic data collapseNextWorld models and cause and effect

All 79 hard problems · Working on this, or know of a result we missed? basant@cleandataengineering.ai