Home › Hard problems › Data and computer systems

Data and computer systems

Keeping thousands of GPUs busy

Solo: possibleOne person can test a narrow slice of it.

GPUz z zGPUz z zGPUz z zGPUz z zbad GPUstuck!GPUz z zGPUz z zGPUz z zthe network: every step waits until all GPUs report backA failureone machine dies, and all goback to the last saved pointA slow machineone straggler, andeveryone waits for itThe networkGPUs sit idle waitingfor each other's data
A big training run moves at the pace of its slowest GPU. One failure, one slow machine or one slow link leaves thousands of others idle.

Why it is still open

Large training runs lose a lot of time to failures, slow machines and waiting on the network. A single bad GPU can stall a whole run.

An experiment one person can run in 8 weeks

Simulate a 1,000-GPU job with realistic failure rates and compare checkpoint strategies on time lost.

Verdict: Solo: possible

Simulation is solo-friendly; real proof needs a cluster.

The solo test behind the verdict

  1. Can it be measured? There is a number that shows you did better, such as an error rate, a catch rate or a cost per answer.
  2. Is it cheap to try? It runs on a laptop, free open models and open data, or under about ₹50,000 of rented GPU time.
  3. Is there a public yardstick? A benchmark or dataset others use, so your result can be compared and believed.
  4. Would a small win get noticed? A lab, a funder or a customer would care about a 10% improvement, not only a full solution.

Four yeses means Solo: strong. Two or three means Solo: possible. Fewer means Follow, don't build.

PreviousEstimating query sizes (the 40-year database problem)NextProving AI-written code is correct

All 79 hard problems · Working on this, or know of a result we missed? basant@cleandataengineering.ai