Home › Hard problems › Safety and oversight

Safety and oversight: checking AI that may be smarter than you

Scalable oversight: weaker people supervising stronger AI

Solo: strongOne person can run a real test of this in about 8 weeks.

the stronger AIa long answer it wrotea hidden mistake? ? ?"Looks right"a thumbs up,so it gets the rewardit learns to look right, not to be right
People can't check answers they don't understand, so training ends up rewarding what looks right.

Why it is still open

People can't check answers they don't understand, so training rewards what looks right.

An experiment one person can run in 8 weeks

Have a small model judge a large model on questions with known answers. Plot how the judge's accuracy falls as the gap widens, then test whether debate or critiques help.

Verdict: Solo: strong

Open models of many sizes make the gap easy to vary.

The solo test behind the verdict

  1. Can it be measured? There is a number that shows you did better, such as an error rate, a catch rate or a cost per answer.
  2. Is it cheap to try? It runs on a laptop, free open models and open data, or under about ₹50,000 of rented GPU time.
  3. Is there a public yardstick? A benchmark or dataset others use, so your result can be compared and believed.
  4. Would a small win get noticed? A lab, a funder or a customer would care about a 10% improvement, not only a full solution.

Four yeses means Solo: strong. Two or three means Solo: possible. Fewer means Follow, don't build.

NextControl: staying safe even if the AI isn't trustworthy

All 79 hard problems · Working on this, or know of a result we missed? basant@cleandataengineering.ai