Home › Hard problems › Safety and oversight

Safety and oversight: checking AI that may be smarter than you

Control: staying safe even if the AI isn't trustworthy

Solo: strongOne person can run a real test of this in about 8 weeks.

a strong agent: it might work around youread logsrun teststidy old files(really: deletes backups)a small trusted monitor:weaker, but on your sideThe real systemfiles, money,codeit has to stop the one bad action hidden among many normal ones,without blocking the good work
Control means staying safe even if the AI is working around you, not just making honest mistakes.

Why it is still open

You need protection against a system that might work around you, not just one that makes honest mistakes.

An experiment one person can run in 8 weeks

Build a trusted monitor: a small model watches a stronger agent's actions and blocks risky ones. Plant sabotage and measure the catch rate and the false alarms.

Verdict: Solo: strong

Mostly systems engineering, and a young field.

The solo test behind the verdict

  1. Can it be measured? There is a number that shows you did better, such as an error rate, a catch rate or a cost per answer.
  2. Is it cheap to try? It runs on a laptop, free open models and open data, or under about ₹50,000 of rented GPU time.
  3. Is there a public yardstick? A benchmark or dataset others use, so your result can be compared and believed.
  4. Would a small win get noticed? A lab, a funder or a customer would care about a 10% improvement, not only a full solution.

Four yeses means Solo: strong. Two or three means Solo: possible. Fewer means Follow, don't build.

PreviousScalable oversight: weaker people supervising stronger AINextReward hacking: the model games the scorer

All 79 hard problems · Working on this, or know of a result we missed? basant@cleandataengineering.ai