Home › Hard problems › Edge AI and serving cost

Edge AI and serving cost: AI on small devices and cheaper answers

Shrinking models below 4 bits without losing quality

Solo: strongOne person can run a real test of this in about 8 weeks.

the same weights, stored with fewer and fewer steps16 bitssmoothfine4 bits16 stepsnearly as good2 bits4 stepsdrops sharplyternaryonly −1, 0, +1drops sharplyreasoning and non-English text are the first to break
Shrinking a model works down to about 4 bits. Below that the weights become a few crude steps, and quality falls off a cliff.

Why it is still open

At 2 bits or "ternary" weights (−1, 0, +1), quality usually drops sharply, especially for reasoning and non-English text.

An experiment one person can run in 8 weeks

Quantize 3 open models to 4, 3 and 2 bits with 3 methods. Measure the quality loss per task type.

Verdict: Solo: strong

Runs on one GPU and fills a measurement gap.

The solo test behind the verdict

  1. Can it be measured? There is a number that shows you did better, such as an error rate, a catch rate or a cost per answer.
  2. Is it cheap to try? It runs on a laptop, free open models and open data, or under about ₹50,000 of rented GPU time.
  3. Is there a public yardstick? A benchmark or dataset others use, so your result can be compared and believed.
  4. Would a small win get noticed? A lab, a funder or a customer would care about a 10% improvement, not only a full solution.

Four yeses means Solo: strong. Two or three means Solo: possible. Fewer means Follow, don't build.

NextThe memory cost of long conversations (the KV cache)

All 79 hard problems · Working on this, or know of a result we missed? basant@cleandataengineering.ai