Home › Hard problems › Edge AI and serving cost

Edge AI and serving cost: AI on small devices and cheaper answers

The memory cost of long conversations (the KV cache)

Solo: strongOne person can run a real test of this in about 8 weeks.

a chat gets longer:every token is keptGPU memorythe model's weightsuser 1user 2user 3user 4user 5Computestill has roomto spareOut of memorynew chats must waitor be turned away
Every token of every conversation is kept in GPU memory while the model answers, so long chats and many users run out of memory before they run out of compute.

Why it is still open

Every token of context is kept in GPU memory while answering, so long chats and many users run out of memory before compute.

An experiment one person can run in 8 weeks

Compare 3 cache-compression methods on one open model at 32k context. Measure memory saved against accuracy lost.

Verdict: Solo: strong

Single-GPU experiment with a direct cost story.

The solo test behind the verdict

  1. Can it be measured? There is a number that shows you did better, such as an error rate, a catch rate or a cost per answer.
  2. Is it cheap to try? It runs on a laptop, free open models and open data, or under about ₹50,000 of rented GPU time.
  3. Is there a public yardstick? A benchmark or dataset others use, so your result can be compared and believed.
  4. Would a small win get noticed? A lab, a funder or a customer would care about a 10% improvement, not only a full solution.

Four yeses means Solo: strong. Two or three means Solo: possible. Fewer means Follow, don't build.

PreviousShrinking models below 4 bits without losing qualityNextAlways-on sensing on tiny chips (TinyML)

All 79 hard problems · Working on this, or know of a result we missed? basant@cleandataengineering.ai