Home › How we work › Estimation mistakes
How we work · free guide · one page
Estimation mistakes in data projects, and how to avoid them
Data projects are famous for running late. Usually the team did not work slowly; the estimate left out most of the work. This page shows why data estimates come in too low, the twelve mistakes we see most, and a way to estimate that holds up.
In plain words
Why data estimates go wrong
In most software work, you can see the thing you are building. In data work, the hardest part is something you have not seen yet: the source data and what it means. Until someone has looked at it, every estimate is a guess about a stranger. And the part people can picture, the code, is the smallest part of the job; we show that on where data engineering time goes.
Two more things make it worse. Effort and calendar time are different: a lot of a data project is waiting for someone else. And the honest range is wide at the start and narrows only as you learn.
Ten days of work can take six weeks. Plan the waits, and start them early.
An honest estimate starts wide and narrows at each step. Promising a single number at the idea stage is promising a guess.
The mistakes
Twelve estimation mistakes and how to avoid them
| Mistake | What happens | How to avoid it |
|---|---|---|
| Estimating the code, not the understanding | The estimate covers ingestion and transforms. Understanding the source, which takes more hours than code, is missing. | Make discovery and source meaning their own line items. See where data engineering time goes. |
| Estimating a source nobody has looked at | A number is given from the source's name ('it's just SAP'). The surprises arrive after commitment. | Give a wide range at idea stage and a firm one only after a time-boxed look at the real data and docs. |
| One number instead of a range | 'Eight weeks' becomes a promise. Every unknown later reads as a slip. | Always a range with the assumptions it rests on, and the event that will narrow it (for example, after the first source runs end to end). |
| Effort and calendar mixed up | Ten days of work is planned as two weeks, then access, answers and sign-offs stretch it to six. | Estimate effort, then add waits separately. Start access and questions on day one. |
| Only the happy path | Late files, duplicates, deletes, corrections and schema changes are not in any line item. | For each source, list how it fails and budget handling, quarantine and tests for each. |
| Reconciliation left out | Numbers are expected to match the old report on the first try. They never do. | Put reconciliation with the business, and the fixes it causes, in as a line item with an owner and a tolerance. |
| History forgotten | The pipeline handles today's data; loading five years of history, with its old formats, is found later. | Ask how far back the data must go and estimate the backfill on its own. |
| Environments and deployment free | Dev works; test, production, CI/CD, secrets and permissions take another two weeks. | Estimate the path to production for the first source; later sources reuse it. |
| Every source the same size | Twenty sources times one average gives a neat number that is wrong for the three hard ones. | Size each source by complexity signals (keys, change column, docs, owner, volume) and estimate per size. |
| Nothing for running it | The estimate ends at go-live; support, reruns and cost tuning in the first weeks come out of the next project. | Add a support period after go-live and say who runs it after that. |
| The estimator is not the doer | A lead or salesperson estimates; the engineers who know the traps never see the number. | The people who build it estimate it, with someone who has done a similar source checking it. |
| Never re-estimating | The first guess stays the plan until the deadline is missed. | Re-estimate at each gate (after discovery, after the first source) and tell people the new range straight away. |
A worked estimate
One medium source, estimated honestly
An order system: about twelve tables, a daily batch, history kept with SCD2, a few gold tables and one dashboard that must match the finance report. The solid bars are what a code-only estimate covers. The striped bars are what usually gets forgotten.
Each bar runs from the low to the high estimate. Hover a bar for its range.
Show the numbers as a table
| Line item | Low (days) | High (days) | Usually in the estimate? |
|---|---|---|---|
| Discovery: what the source data means | 12 | 18 | no, usually forgotten |
| Access and environment setup (effort only) | 2 | 3 | no, usually forgotten |
| Ingestion to bronze | 4 | 6 | yes |
| Bronze to silver, with SCD2 history | 8 | 12 | yes |
| Gold tables and metric definitions | 6 | 10 | yes |
| Tests: unit, contract, reconciliation checks | 6 | 9 | no, usually forgotten |
| Reconciling with finance and fixing | 8 | 14 | no, usually forgotten |
| Loading history (backfill) | 3 | 5 | no, usually forgotten |
| Deploy, CI/CD, monitoring, runbook | 4 | 6 | no, usually forgotten |
| Support after go-live (first 4 weeks) | 5 | 8 | no, usually forgotten |
| Total | 58 | 91 |
The code-only estimate says 18 to 28 days. The honest one says 58 to 91 days, roughly 12 to 18 person-weeks, plus the waits for access and answers on the calendar. Nobody worked slowly; the first number just left most of the work out.
Many sources
Size each source before you add them up
Twenty sources are never twenty copies of an average source. Put each one in a size by what you can see about it, then use a range per size.
| Source size | Signals | First build, person-weeks |
|---|---|---|
| Small | A few tables, clear keys, a reliable change column, docs exist, owner answers within a day | 1 to 2 |
| Medium | Ten to thirty tables, some joins to learn, deletes need care, some docs, owner busy | 3 to 5 |
| Large | Many tables or a packaged app (ERP, CRM), codes that need lookup, no change column, history in odd formats | 6 to 10 |
| Very large | Several systems feeding one model, conflicting definitions, streaming plus batch, no reachable owner | 10 to 20, split it |
Choosing
Ways to estimate, with verdicts
| Way to estimate | How it works | Verdict |
|---|---|---|
| Gut feel, one number | Someone senior says a number in a meeting. | Drop: no assumptions to check, no way to narrow it |
| Task list, bottom up | List every task and add the days. | Pursue with care: good once the source is known, but misses what nobody thought to list |
| Size by source complexity | Put each source in a size (small to very large) by its signals, use an effort range per size. | Pursue: the main method for many sources |
| Compare with past projects | Look at how long similar sources really took last time, not how long they were estimated. | Pursue: the best check on any estimate; start keeping the records now |
| Time-boxed look first | Spend two to five days on the real data and docs, then estimate. | Pursue: cheap, and removes the biggest unknowns |
| Fixed price before discovery | Commit to a cost before anyone has seen the data. | Drop, or split: fix the price of discovery only, then estimate the rest |
Your turn
Five questions for your next estimate
- Does your estimate have a line for understanding the source data, and is it bigger than the code line?
- Is it a range, and does it say what event will narrow the range?
- Are waits (access, answers, sign-off) shown apart from effort?
- Is reconciliation in, with a tolerance and someone to sign it off?
- Did the people who will build it make the estimate?
FAQ
Questions people ask
Why are data project estimates usually too low?
Because people estimate the code they can picture, and the code is the smallest part. Understanding source data, reconciling numbers, waiting for access, loading history and support after go-live are left out. In our worked example the usual code-only estimate is about a third of the honest one.
How do you estimate a data migration or a new source?
Spend a few days looking at the real data and documents first. Size each source by its signals: number of tables, whether keys and a change column exist, whether docs and an owner are available. Use a range per size, add reconciliation, backfill and support, and show waits apart from effort.
Should a data project estimate be a range?
Yes. A single number hides the unknowns and becomes a promise. Give a range, the assumptions behind it, and the point at which you will narrow it, usually after the first source runs end to end.
How much contingency should a data project have?
Rather than a flat percentage on top, put the usually forgotten items in as real line items. Then keep a contingency for the sources you have not looked at yet, and shrink it as you look.
Are the numbers on this page measured?
No. They are rounded field estimates from data platform projects, to show the shape. Your own past projects are the best source: record how long each source really took.
Related
Where it connects
Where data engineering time goes · Project management mistakes · Critical thinking in data projects · Discovery · Source system questionnaire · How we work