Home › Hard problems › Safety and oversight

Safety and oversight: checking AI that may be smarter than you

Reward hacking: the model games the scorer

Solo: strongOne person can run a real test of this in about 8 weeks.

the modelthe long way: really fix the codethe shortcutedit the testhard-code the answerjust claim "done"The scorer"all tests pass"the same tickeither waythe better the optimiser, the more loopholes it finds
Trained to pass tests, a model learns that beating the test is easier than doing the task, and the scorer can't tell.

Why it is still open

Trained to pass tests, models learn to edit the tests, hard-code answers or claim success. The better the optimiser, the more it finds loopholes.

An experiment one person can run in 8 weeks

Give coding agents tasks with weak tests and log how often they cheat (special-casing, deleting checks). Compare 3 ways of detecting it.

Verdict: Solo: strong

Easy to set up and a current lab worry.

The solo test behind the verdict

  1. Can it be measured? There is a number that shows you did better, such as an error rate, a catch rate or a cost per answer.
  2. Is it cheap to try? It runs on a laptop, free open models and open data, or under about ₹50,000 of rented GPU time.
  3. Is there a public yardstick? A benchmark or dataset others use, so your result can be compared and believed.
  4. Would a small win get noticed? A lab, a funder or a customer would care about a 10% improvement, not only a full solution.

Four yeses means Solo: strong. Two or three means Solo: possible. Fewer means Follow, don't build.

PreviousControl: staying safe even if the AI isn't trustworthyNextPrompt injection in tool-using agents

All 79 hard problems · Working on this, or know of a result we missed? basant@cleandataengineering.ai