Home › Hard problems › Data and computer systems

Data and computer systems

Useful analytics with privacy guarantees

Solo: possibleOne person can test a narrow slice of it.

THE TRUE COUNTADD NOISEWHAT IS PUBLISHEDSay, 1,000,000people in a stateAbout 1,000,030still usefulthe noise is tinynext to the countSay, 12people at one clinicCould be 0, could be 60useless for a decisionthe same noiseswamps the countsay, plus or minus 50, so no one person can be picked outhidden, butthe numberis lost too
Differential privacy adds random noise so no person can be identified. The noise barely touches big groups but makes small groups' numbers useless.

Why it is still open

Differential privacy (adding maths noise so no person can be identified) makes small groups' numbers useless.

An experiment one person can run in 8 weeks

Apply it to a public census-style table and measure which questions stay accurate enough to act on.

Verdict: Solo: possible

Clear numbers; the buyers are governments and health bodies.

The solo test behind the verdict

  1. Can it be measured? There is a number that shows you did better, such as an error rate, a catch rate or a cost per answer.
  2. Is it cheap to try? It runs on a laptop, free open models and open data, or under about ₹50,000 of rented GPU time.
  3. Is there a public yardstick? A benchmark or dataset others use, so your result can be compared and believed.
  4. Would a small win get noticed? A lab, a funder or a customer would care about a 10% improvement, not only a full solution.

Four yeses means Solo: strong. Two or three means Solo: possible. Fewer means Follow, don't build.

PreviousMatching records that refer to the same thingNextData meaning, not just data movement

All 79 hard problems · Working on this, or know of a result we missed? basant@cleandataengineering.ai