Home › Hard problems › Inside the model

Inside the model: the core of LLMs

Continual learning: learn after deployment without forgetting

Solo: possibleOne person can test a narrow slice of it.

THE MODEL'S WEIGHTS: ONE FULL SHELFnew facts,this weekupdate theweightsold skillsfall off("catastrophicforgetting")Today's fixretrain from scratch, or bolt on retrieval
Teaching a shipped model new facts by updating its weights knocks old skills off the shelf.

Why it is still open

Updating weights on new facts damages old skills ("catastrophic forgetting"). Today's fix is retraining from scratch or bolting on retrieval.

An experiment one person can run in 8 weeks

Fine-tune a small model (1–3B) on a stream of weekly new facts. Measure how much old skill is lost per update, then compare 3 known methods (replay, LoRA adapters per period, model merging).

Verdict: Solo: possible

Small models make it feasible. A clean comparison is the goal, not beating labs.

The solo test behind the verdict

  1. Can it be measured? There is a number that shows you did better, such as an error rate, a catch rate or a cost per answer.
  2. Is it cheap to try? It runs on a laptop, free open models and open data, or under about ₹50,000 of rented GPU time.
  3. Is there a public yardstick? A benchmark or dataset others use, so your result can be compared and believed.
  4. Would a small win get noticed? A lab, a funder or a customer would care about a 10% improvement, not only a full solution.

Four yeses means Solo: strong. Two or three means Solo: possible. Fewer means Follow, don't build.

PreviousThinking cost: models overthink easy questionsNextUsing long context, not just fitting it

All 79 hard problems · Working on this, or know of a result we missed? basant@cleandataengineering.ai