Home › How we work › Estimation mistakes

How we work · free guide · one page

Estimation mistakes in data projects, and how to avoid them

Data projects are famous for running late. Usually the team did not work slowly; the estimate left out most of the work. This page shows why data estimates come in too low, the twelve mistakes we see most, and a way to estimate that holds up.

18–28person-days: the usual code-only estimate for one medium source.
58–91person-days: the same source with everything in. About three times as much.
6 weeksof calendar for 10 days of work, once waits are counted.
Read this first: the figures are rounded field estimates from data platform projects, there to show the shape, not a measured study. Your own past projects are the best numbers you have.

In plain words

Why data estimates go wrong

In most software work, you can see the thing you are building. In data work, the hardest part is something you have not seen yet: the source data and what it means. Until someone has looked at it, every estimate is a guess about a stranger. And the part people can picture, the code, is the smallest part of the job; we show that on where data engineering time goes.

Two more things make it worse. Effort and calendar time are different: a lot of a data project is waiting for someone else. And the honest range is wide at the start and narrows only as you learn.

effortcalendar10 days of real workwaiting for accesswaiting for answerswaiting for sign-offsix weeksestimate effort and calendar separately; waiting costs time, not effort

Ten days of work can take six weeks. Plan the waits, and start them early.

ideaafter discoveryone source end to endhalf the sourcesall liveactualhow wide the honest range is, as the project learns (illustrative)

An honest estimate starts wide and narrows at each step. Promising a single number at the idea stage is promising a guess.

The mistakes

Twelve estimation mistakes and how to avoid them

MistakeWhat happensHow to avoid it
Estimating the code, not the understandingThe estimate covers ingestion and transforms. Understanding the source, which takes more hours than code, is missing.Make discovery and source meaning their own line items. See where data engineering time goes.
Estimating a source nobody has looked atA number is given from the source's name ('it's just SAP'). The surprises arrive after commitment.Give a wide range at idea stage and a firm one only after a time-boxed look at the real data and docs.
One number instead of a range'Eight weeks' becomes a promise. Every unknown later reads as a slip.Always a range with the assumptions it rests on, and the event that will narrow it (for example, after the first source runs end to end).
Effort and calendar mixed upTen days of work is planned as two weeks, then access, answers and sign-offs stretch it to six.Estimate effort, then add waits separately. Start access and questions on day one.
Only the happy pathLate files, duplicates, deletes, corrections and schema changes are not in any line item.For each source, list how it fails and budget handling, quarantine and tests for each.
Reconciliation left outNumbers are expected to match the old report on the first try. They never do.Put reconciliation with the business, and the fixes it causes, in as a line item with an owner and a tolerance.
History forgottenThe pipeline handles today's data; loading five years of history, with its old formats, is found later.Ask how far back the data must go and estimate the backfill on its own.
Environments and deployment freeDev works; test, production, CI/CD, secrets and permissions take another two weeks.Estimate the path to production for the first source; later sources reuse it.
Every source the same sizeTwenty sources times one average gives a neat number that is wrong for the three hard ones.Size each source by complexity signals (keys, change column, docs, owner, volume) and estimate per size.
Nothing for running itThe estimate ends at go-live; support, reruns and cost tuning in the first weeks come out of the next project.Add a support period after go-live and say who runs it after that.
The estimator is not the doerA lead or salesperson estimates; the engineers who know the traps never see the number.The people who build it estimate it, with someone who has done a similar source checking it.
Never re-estimatingThe first guess stays the plan until the deadline is missed.Re-estimate at each gate (after discovery, after the first source) and tell people the new range straight away.

A worked estimate

One medium source, estimated honestly

An order system: about twelve tables, a daily batch, history kept with SCD2, a few gold tables and one dashboard that must match the finance report. The solid bars are what a code-only estimate covers. The striped bars are what usually gets forgotten.

Worked estimate, one medium source, person-days (low to high)usually in the estimateusually forgottenDiscovery: what the source data meansDiscovery: what the source data means: 12 to 18 person-days12–18Access and environment setup (effort only)Access and environment setup (effort only): 2 to 3 person-days2–3Ingestion to bronzeIngestion to bronze: 4 to 6 person-days4–6Bronze to silver, with SCD2 historyBronze to silver, with SCD2 history: 8 to 12 person-days8–12Gold tables and metric definitionsGold tables and metric definitions: 6 to 10 person-days6–10Tests: unit, contract, reconciliation checksTests: unit, contract, reconciliation checks: 6 to 9 person-days6–9Reconciling with finance and fixingReconciling with finance and fixing: 8 to 14 person-days8–14Loading history (backfill)Loading history (backfill): 3 to 5 person-days3–5Deploy, CI/CD, monitoring, runbookDeploy, CI/CD, monitoring, runbook: 4 to 6 person-days4–6Support after go-live (first 4 weeks)Support after go-live (first 4 weeks): 5 to 8 person-days5–805101520

Each bar runs from the low to the high estimate. Hover a bar for its range.

Show the numbers as a table
Line itemLow (days)High (days)Usually in the estimate?
Discovery: what the source data means1218no, usually forgotten
Access and environment setup (effort only)23no, usually forgotten
Ingestion to bronze46yes
Bronze to silver, with SCD2 history812yes
Gold tables and metric definitions610yes
Tests: unit, contract, reconciliation checks69no, usually forgotten
Reconciling with finance and fixing814no, usually forgotten
Loading history (backfill)35no, usually forgotten
Deploy, CI/CD, monitoring, runbook46no, usually forgotten
Support after go-live (first 4 weeks)58no, usually forgotten
Total5891

The code-only estimate says 18 to 28 days. The honest one says 58 to 91 days, roughly 12 to 18 person-weeks, plus the waits for access and answers on the calendar. Nobody worked slowly; the first number just left most of the work out.

Many sources

Size each source before you add them up

Twenty sources are never twenty copies of an average source. Put each one in a size by what you can see about it, then use a range per size.

Source sizeSignalsFirst build, person-weeks
SmallA few tables, clear keys, a reliable change column, docs exist, owner answers within a day1 to 2
MediumTen to thirty tables, some joins to learn, deletes need care, some docs, owner busy3 to 5
LargeMany tables or a packaged app (ERP, CRM), codes that need lookup, no change column, history in odd formats6 to 10
Very largeSeveral systems feeding one model, conflicting definitions, streaming plus batch, no reachable owner10 to 20, split it

Choosing

Ways to estimate, with verdicts

Way to estimateHow it worksVerdict
Gut feel, one numberSomeone senior says a number in a meeting.Drop: no assumptions to check, no way to narrow it
Task list, bottom upList every task and add the days.Pursue with care: good once the source is known, but misses what nobody thought to list
Size by source complexityPut each source in a size (small to very large) by its signals, use an effort range per size.Pursue: the main method for many sources
Compare with past projectsLook at how long similar sources really took last time, not how long they were estimated.Pursue: the best check on any estimate; start keeping the records now
Time-boxed look firstSpend two to five days on the real data and docs, then estimate.Pursue: cheap, and removes the biggest unknowns
Fixed price before discoveryCommit to a cost before anyone has seen the data.Drop, or split: fix the price of discovery only, then estimate the rest
How to hand an estimate over: a range, the assumptions behind it, the items that are usually forgotten shown on their own lines, the waits shown apart from effort, and the date you will re-estimate.

Your turn

Five questions for your next estimate

  1. Does your estimate have a line for understanding the source data, and is it bigger than the code line?
  2. Is it a range, and does it say what event will narrow the range?
  3. Are waits (access, answers, sign-off) shown apart from effort?
  4. Is reconciliation in, with a tolerance and someone to sign it off?
  5. Did the people who will build it make the estimate?

FAQ

Questions people ask

Why are data project estimates usually too low?

Because people estimate the code they can picture, and the code is the smallest part. Understanding source data, reconciling numbers, waiting for access, loading history and support after go-live are left out. In our worked example the usual code-only estimate is about a third of the honest one.

How do you estimate a data migration or a new source?

Spend a few days looking at the real data and documents first. Size each source by its signals: number of tables, whether keys and a change column exist, whether docs and an owner are available. Use a range per size, add reconciliation, backfill and support, and show waits apart from effort.

Should a data project estimate be a range?

Yes. A single number hides the unknowns and becomes a promise. Give a range, the assumptions behind it, and the point at which you will narrow it, usually after the first source runs end to end.

How much contingency should a data project have?

Rather than a flat percentage on top, put the usually forgotten items in as real line items. Then keep a contingency for the sources you have not looked at yet, and shrink it as you look.

Are the numbers on this page measured?

No. They are rounded field estimates from data platform projects, to show the shape. Your own past projects are the best source: record how long each source really took.

Related

Where it connects

Where data engineering time goes · Project management mistakes · Critical thinking in data projects · Discovery · Source system questionnaire · How we work