Home › Hard problems › Data and computer systems

Data and computer systems

Data meaning, not just data movement

Solo: possibleOne person can test a narrow slice of it.

Raw ordersrevenuesay: includes taxrevenuesay: after refundsrevenuesay: booked, not yet paidevery copy moves correctlyDashboard"revenue":three numbersMeaning checknobody has onewhich revenue is real?
Pipelines move data correctly, yet "revenue" can mean three things in three tables, and nothing checks the meaning.

Why it is still open

Pipelines move data correctly, yet "revenue" means three different things in three tables, and nothing checks the meaning.

An experiment one person can run in 8 weeks

Build a checker that compares metric definitions across dbt models and dashboards, and flags conflicts on an open project.

Verdict: Solo: possible

Useful, but harder to state as one number.

The solo test behind the verdict

  1. Can it be measured? There is a number that shows you did better, such as an error rate, a catch rate or a cost per answer.
  2. Is it cheap to try? It runs on a laptop, free open models and open data, or under about ₹50,000 of rented GPU time.
  3. Is there a public yardstick? A benchmark or dataset others use, so your result can be compared and believed.
  4. Would a small win get noticed? A lab, a funder or a customer would care about a 10% improvement, not only a full solution.

Four yeses means Solo: strong. Two or three means Solo: possible. Fewer means Follow, don't build.

PreviousUseful analytics with privacy guarantees

All 79 hard problems · Working on this, or know of a result we missed? basant@cleandataengineering.ai