Home › How we work › Critical thinking
How we work · free guide · one page
Critical thinking in data projects: five questions for every claim
A data project runs on claims. The source team says the key is unique. A vendor says their engine is five times faster. A dashboard says sales fell. A requirement says users need it in real time. Critical thinking, here, means one plain thing: check a claim before you build on it. This page gives five questions to ask, shows where claims hide at each stage of a project, and lists the traps numbers set.
The method
Five questions for every claim
You do not need all five every time. Use them as a checklist when a claim matters: when a design, a number in a report, or a promise to a user rests on it.
Five questions. The fifth decides how hard to push on the other four.
1. What exactly is being claimed?
Write the claim as one sentence with numbers and a scope. "The data is clean" is not a claim anyone can check. "Fewer than 0.1% of order rows since January have no customer id" is. Half of all arguments in a project end here, because people find they meant different things.
2. What is the evidence, and who measured it?
Ask how the person knows. Did they run a query, read a document, hear it in a meeting, or remember it? Who measured it, on what data, when, and do they gain if it is true? A vendor's benchmark, a source team's spec and last year's profiling are all evidence, but of different strength.
3. What else could explain it?
Before accepting a cause, list two other causes that would produce the same numbers. Sales dropped after the release; so did the number of rows loaded, because a file was late. If you cannot name another explanation, you have not looked yet.
4. What would prove it wrong, and can we test that cheaply?
Name the observation that would break the claim, then look for it. "No duplicates" breaks if one key appears twice: a single GROUP BY finds out in seconds. Looking for the counter-example is faster and more honest than collecting examples that agree.
5. What does it cost if we are wrong?
Not every claim deserves a week of checking. If being wrong costs an hour of rework, accept it and move on. If it costs a wrong number in a board report or a regulator's file, check it hard. Match the effort to the price of the mistake.
Where to look
Where claims hide in a data project
Every stage has its favourite claims. Each one below sounds reasonable, and each has gone wrong on real projects. The right-hand column is the cheap check: usually one query or one question.
| Stage | Typical claim | The question to ask | A cheap check |
|---|---|---|---|
| Discovery | "Users need this data daily." | Who said it, and what decision do they make with it each day? | Ask for the last three times they needed it and what they did. |
| Source onboarding | "The key is unique; deletes are flagged." | Says who: the doc, the DBA, or the data? | Profile a month of data: count keys, look for rows that vanish between loads. |
| Design | "Streaming is needed; batch is too slow." | Too slow for which decision, by how many minutes? | Ask what happens today when the number is an hour old. |
| Build | "This join is correct." | What would a wrong join look like? | Compare row counts before and after the join; check one known customer by hand. |
| Testing | "All tests pass." | Would these tests fail if the code were wrong? | Break the code on purpose once and see whether a test catches it. |
| Go-live | "Numbers match the old report." | Match on which days, at what tolerance, and what about the days that don't? | Reconcile a bad month, not just a quiet one. |
| Running | "The pipeline is healthy; it is green." | Green means it ran. Did it load the right rows? | Track rows in and rows out per load, not just job status. |
Traps with numbers
Seven ways numbers mislead
Numbers do not lie, but they leave things out. These are the traps we meet most often.
Twenty-six quiet nights and three bad ones. The average sits where no night ever was.
| Trap | What it looks like | What to do |
|---|---|---|
| An average hides the spread | "Loads take 5 minutes on average." Most take 2 or 3; three nights took about 30. Plan for the 30. | Look at the spread: the median, the slowest 5%, and a picture of every value. |
| Only the survivors were looked at | "Every load this month had valid dates." The loads that failed were never counted. | Ask what was excluded, and count the rejected and quarantined rows too. |
| Moved together is not caused | "The new index made queries faster." The cluster was also resized that day. | Change one thing at a time, or compare against a group that did not change. |
| The total disagrees with every part | Plant B is better for every tyre type, yet worse overall (worked example below). | Split by the obvious groups before comparing totals. |
| The denominator changed | "Error rate fell by half." The count of records checked doubled when a new source was added. | Always show both numbers of a rate, and say what was counted. |
| The benchmark compared unlike things | "Engine X is 5 times faster." On a different data size, file layout and warm cache. | Re-run on your own data shape, same size machines, cold and warm. |
| "The source document says" | The document was written for version 3; the system is on version 5. | Treat documents as claims to test, and date every answer you write down. |
A worked example: the total that disagrees with every part
Two plants make car and truck tyres. Management asks which plant has fewer returns. The overall numbers say Plant A, clearly. Split by tyre type and Plant B wins both times. Both statements are true. Plant B makes mostly truck tyres, which come back more often whoever makes them, so its total is dragged up by its mix, not its quality. This is called Simpson's paradox. The numbers are made up to teach the idea.
Plant B is better for car tyres and for truck tyres, yet looks worse overall.
| Plant | Tyre type | Sold | Returned | Return rate |
|---|---|---|---|---|
| Plant A | Car | 900 | 18 | 2.0% |
| Plant A | Truck | 100 | 8 | 8.0% |
| Plant B | Car | 200 | 3 | 1.5% |
| Plant B | Truck | 800 | 56 | 7.0% |
| Plant A | All | 1,000 | 26 | 2.6% |
| Plant B | All | 1,000 | 59 | 5.9% |
The lesson for any comparison: before comparing totals, split by the obvious groups (product, region, channel, source system) and check the mix is similar.
In practice
Three claims, checked
"The source has no duplicates"
The source team said so in the first meeting, and the design used order_id as the key. Question four asked what would prove it wrong: one order_id twice. A GROUP BY over three months found 412 keys appearing twice, all on days when the source system was restarted mid-batch. The claim was true for normal days and false on bad ones. The fix cost one dedupe rule in silver, written before go-live instead of after the first wrong report.
"Engine X is 5 times faster"
A vendor slide showed a query running five times faster on their engine. Question two asked who measured it and on what: the vendor, on a tidy data set ten times smaller than ours, with a warm cache. Question five asked the cost of being wrong: a platform choice for five years. So the team ran three of its own slowest queries on both engines at the same size and price. The gap was real but smaller, and on one query it went the other way. The decision was made on our numbers, written in the decision log.
"Sales dropped 12% after the release"
A dashboard showed sales down 12% the day after a pipeline release, and the release was blamed. Question three asked what else could explain it. Rows loaded were also down 12%: one region's file arrived after the load ran. Sales had not dropped; the data had. The release was fine. The lasting fix was a freshness check per region, so a late file shows as "incomplete" on the dashboard instead of as a fall in sales.
For teams
Habits that make it routine
Critical thinking works best as a team habit, not one sceptical person. Start by labelling what you write down, so everyone can see which parts are known and which are hoped.
Three labels for anything in a design or decision log. An assumption always has an owner who will check it.
| Habit | What it means | Verdict | Why |
|---|---|---|---|
| Write the claim and its evidence in the decision log | Every decision lists the claims it rests on, each marked fact, assumption or opinion, with a link to the query or source. | Pursue | Cheap, and the next person can check the reasoning instead of guessing it. |
| Ask "how do we know?" in every review | One person's job in each design review is to ask it of every key claim. | Pursue | Turns doubt into a routine, so it never feels personal. |
| Pre-mortem before a big build | Imagine it is six months later and the project failed. Each person writes why. Turn the top reasons into checks. | Pursue | Finds the risks people were too polite to raise. |
| Red-team review for big decisions | Someone argues the other side on purpose, with evidence. | Pursue for decisions that are hard to undo | Too slow for small choices; worth it for the platform and the format. |
| Test the counter-example first | Before collecting evidence for a claim, write the one query that would prove it wrong, and run it. | Pursue | Usually a few minutes, and it catches the most expensive mistakes. |
| Check everything to the same depth | Every claim gets a full investigation. | Drop | Burns time on cheap mistakes; use question five to choose. |
| Trust the most senior voice | The most experienced person's view settles the question. | Drop | Experience is evidence, but of one kind; it still has to meet the data. |
AI answers
Using AI critically
- An answer from an AI assistant is a claim like any other. Ask the same five questions of it.
- It writes with the same confidence whether it is right or wrong, so its tone is not evidence.
- Ask it for the query, the document or the reasoning behind an answer, then run or read that yourself.
- Use it well: to list other explanations (question three), to draft the counter-example query (question four), and to spot claims in a long document.
- Never let it fill in what a source system means. It can draft the question; a person who knows answers it.
Your turn
Five questions for your own project
- Pick the three claims your current design rests on most. For each, can you point to the query or document that supports it?
- Which of your project's beliefs are still assumptions, and who owns checking each one?
- When did a test last fail because the code was wrong, rather than because the environment broke?
- Does your pipeline report rows in and rows out per load, or only success and failure?
- Which decision in your project would be the most expensive to get wrong, and how hard was it checked?
FAQ
Questions people ask
What is critical thinking in a data project?
Checking a claim before you build on it. In practice, asking five questions: what exactly is claimed, what is the evidence, what else could explain it, what would prove it wrong, and what it costs to be wrong.
Doesn't this slow the project down?
It speeds it up when you match the effort to the cost of being wrong. Most checks are one query or one question. The slow projects are the ones that find a wrong assumption after go-live.
How do I question a senior person's claim without causing offence?
Make it a routine, not a challenge. If every review asks "how do we know?" of every key claim, nobody is singled out. Ask for the evidence, not for their reasons.
What is Simpson's paradox, simply?
A comparison that holds in every group can reverse when the groups are added together, because the groups are mixed in different amounts. Plant B returns fewer tyres of every type but more overall, because it makes mostly truck tyres, which are returned more often.
Should I trust a vendor benchmark?
Treat it as a claim made by someone who gains if it is true. Check the data size, file layout, cache and machine size, then re-run your own slowest queries on equal terms before deciding.
Related
Where it connects
How we work · Estimation mistakes in data projects · Where data engineering time goes · Discovery: fail fast for the whole project · Upstreams: know your sources