Home › Hard problems › Data and computer systems

Data and computer systems

Same input, different LLM answer

Solo: strongOne person can run a real test of this in about 8 weeks.

Same prompttemperature 0,sent 3 timesThe serverbatches it with otherusers' requestsRun 1"approve the refund"Run 2"approve the refund"Run 3"reject the refund"a different batch adds GPU sumsin a different order: tiny roundingchanges can flip a wordwhich one do I audit?
Even at temperature 0, batching and GPU maths can change an LLM's answer between runs, which breaks audits and repeatable tests.

Why it is still open

Even at "temperature 0", batching and GPU maths make answers vary between runs, which breaks audits and repeatable tests.

An experiment one person can run in 8 weeks

Measure how often answers change across runs and providers for 1,000 prompts. Test which serving settings make them repeatable.

Verdict: Solo: strong

Cheap, and regulated buyers need it.

The solo test behind the verdict

  1. Can it be measured? There is a number that shows you did better, such as an error rate, a catch rate or a cost per answer.
  2. Is it cheap to try? It runs on a laptop, free open models and open data, or under about ₹50,000 of rented GPU time.
  3. Is there a public yardstick? A benchmark or dataset others use, so your result can be compared and believed.
  4. Would a small win get noticed? A lab, a funder or a customer would care about a 10% improvement, not only a full solution.

Four yeses means Solo: strong. Two or three means Solo: possible. Fewer means Follow, don't build.

PreviousProving AI-written code is correctNextRetrieval that finds the right passage

All 79 hard problems · Working on this, or know of a result we missed? basant@cleandataengineering.ai