Home › Hard problems › Inside the model

Inside the model: the core of LLMs

New architectures beyond the transformer

Follow, don't buildNeeds labs, hardware or big teams. Learn it and watch it.

TRANSFORMER: EVERY WORD LOOKS AT EVERY OTHER4 words16 checks8 words64 checks16 words256 checksdouble the length,four times the workALTERNATIVES (MAMBA, HYBRIDS)one fixed-sizerunning notecost grows in a straight line: cheaperbut not yet clearly as good?
Transformer cost grows with the square of the context length. Cheaper designs exist, but nobody has shown they are as good.

Why it is still open

Transformer cost grows with the square of context length. Alternatives (state-space models such as Mamba, hybrids) are cheaper but not yet clearly as good.

An experiment one person can run in 8 weeks

Train tiny models of each type on the same data and compare quality per unit of compute on one long-sequence task.

Verdict: Follow, don't build

Real answers need large training runs. Learn it at Explain depth.

The solo test behind the verdict

  1. Can it be measured? There is a number that shows you did better, such as an error rate, a catch rate or a cost per answer.
  2. Is it cheap to try? It runs on a laptop, free open models and open data, or under about ₹50,000 of rented GPU time.
  3. Is there a public yardstick? A benchmark or dataset others use, so your result can be compared and believed.
  4. Would a small win get noticed? A lab, a funder or a customer would care about a 10% improvement, not only a full solution.

Four yeses means Solo: strong. Two or three means Solo: possible. Fewer means Follow, don't build.

PreviousWorld models and cause and effectNextLearning from far less data

All 79 hard problems · Working on this, or know of a result we missed? basant@cleandataengineering.ai