Home › Hard problems › Chips, compute and energy

Chips, compute and energy

The memory wall

Solo: possibleOne person can test a narrow slice of it.

Memoryholds the model,feeds it slowlya slow conveyor brings the numbersChipz z zfast, but itsplate is emptymost of the time
Chips compute far faster than memory can feed them, so serving an LLM is mostly waiting for memory, not doing maths.

Why it is still open

Chips compute far faster than memory can feed them. Serving LLMs is mostly waiting for memory, not maths.

An experiment one person can run in 8 weeks

Profile one open model on one GPU and show what share of time is memory-bound at different batch sizes.

Verdict: Solo: possible

A good learning project and an explainer, not a breakthrough.

The solo test behind the verdict

  1. Can it be measured? There is a number that shows you did better, such as an error rate, a catch rate or a cost per answer.
  2. Is it cheap to try? It runs on a laptop, free open models and open data, or under about ₹50,000 of rented GPU time.
  3. Is there a public yardstick? A benchmark or dataset others use, so your result can be compared and believed.
  4. Would a small win get noticed? A lab, a funder or a customer would care about a 10% improvement, not only a full solution.

Four yeses means Solo: strong. Two or three means Solo: possible. Fewer means Follow, don't build.

NextConnecting thousands of chips

All 79 hard problems · Working on this, or know of a result we missed? basant@cleandataengineering.ai