Home › How we work › Project management mistakes
How we work · free guide · one page
Top project management mistakes in data projects, and how to avoid them
Data projects rarely fail because the code was hard. They fail because the plan treated them like ordinary software projects: a list of things to build, a date, and percent complete each week. This page shows what is different about data projects, the thirteen mistakes we see most, and what to do instead.
Teach first
Why data projects break normal project management
Three things make a data project different from building an app.
- It is discovery as much as delivery. You don't know what the source data really means until you look, ask and test. Some of what you learn changes the plan.
- The biggest dependency is not yours. The source systems, and the people who know them, belong to other teams with their own priorities.
- Done means trusted numbers, not shipped code. A pipeline that runs but disagrees with finance is not done. Proving the numbers is a phase of its own.
A software project is mostly a straight line. A data project loops back each time the numbers don't match, and depends on systems it doesn't own.
Mistakes: planning
Planning
1Planning as if the code were the work
What it looks like: The plan lists ingest, transform, test, deploy, each with days of coding.
Why it hurts: Most hours go to understanding what the source data means, reconciling and waiting. A plan of code tasks is late from week one.
How to avoid it: Plan the understanding work as real tasks: questions per source, answers signed, reconciliation. See where data engineering time goes.
2No discovery phase and no go / no-go gate
What it looks like: The team starts building in week one because the deadline is fixed.
Why it hurts: The project learns in month four that a key source has no change date, or that two systems disagree on the customer. The plan breaks late, when it costs most.
How to avoid it: A short discovery with a written go / no-go decision before the build is funded. See discovery: fail fast for the whole project.
3Estimates given as one number
What it looks like: "Twelve weeks." It goes on a slide and becomes a promise.
Why it hurts: Unknown sources make a single number a guess. When it slips, trust in the whole team drops.
How to avoid it: Give ranges, say what would move them, and narrow them after discovery. See estimation mistakes.
Mistakes: scope and requirements
Scope and requirements
4Scope as "migrate everything"
What it looks like: The old warehouse has 900 tables and reports, so the new one must have all of them.
Why it hurts: Half of them nobody uses. The team spends months rebuilding reports that get switched off anyway.
How to avoid it: List who uses what (query logs help), then move by decision: the reports people act on first. Retire the rest on purpose.
5Requirements as report mock-ups
What it looks like: A picture of the dashboard, with no definition behind each number.
Why it hurts: "Active customer" means three things in three departments. The engineer picks one, and the argument happens at go-live.
How to avoid it: Each number on the mock-up gets a written definition, a worked example and a named owner who signs it.
Mistakes: people and dependencies
People and dependencies
6Source owners not committed
What it looks like: The source team was told about the project, but answering questions is not in their plan.
Why it hurts: Every question waits days. The source is the one dependency the project cannot work around.
How to avoid it: Get a named source owner and an agreed answer time before the build starts. Put it in their plan, not just yours.
7Access requested when coding starts
What it looks like: Week three: the engineer asks for credentials, a firewall rule and a security review.
Why it hurts: Approvals take weeks. The team waits, or works on samples that turn out different from the real data.
How to avoid it: Raise every access request on day one, in parallel. Make "access granted" a gate before the build.
8The engineer as the stakeholder chaser
What it looks like: The engineer runs the meetings, chases answers and writes the status.
Why it hurts: Your most expensive builder spends the week in calls. Answers stay in their head and leave with them.
How to avoid it: The PM or a technical anchor chases and writes answers down; the engineer builds from what is written.
The chasing has to happen. It should not land on the person who builds.
Mistakes: delivery and quality
Delivery and quality
9No reconciliation target or sign-off
What it looks like: "It should match the old report", with no tolerance and no one to sign.
Why it hurts: Every small difference becomes an investigation. Go-live keeps moving because "done" was never defined.
How to avoid it: Agree up front: which report, which tolerance, the known differences, and who signs off. Automate the check.
10Status by percent complete
What it looks like: "Bronze 80%, silver 60%, gold 30%: 54% done."
Why it hurts: Nothing works for a user until the last layer is done, so the percentages hide the real risk.
How to avoid it: Report what works end to end: "orders reach the report and match finance". Build thin slices so that sentence comes early.
11Big-bang go-live with no parallel run
What it looks like: The old system is switched off the day the new one goes live.
Why it hurts: The first month-end close is the first real test, and there is no way back.
How to avoid it: Run old and new side by side for at least one full business cycle, such as a month-end, and compare daily.
Percent complete per layer looks like progress. One source working end to end is progress.
Mistakes: after go-live
After go-live
12No plan for running it
What it looks like: The project ends at go-live. Nobody owns failures, reruns, source changes or the cloud bill.
Why it hurts: Quality fixes and reruns become the biggest share of the work, done by people already moved to the next project.
How to avoid it: Plan the run before go-live: who is on call, how reruns and backfills work, alerts on source changes, a cost budget.
13Success measured as pipelines built
What it looks like: The closing slide counts tables, pipelines and terabytes.
Why it hurts: Nobody checks whether anyone decided anything differently. The next budget is hard to win.
How to avoid it: Name two or three decisions the data should change, at the start. Check them three months after go-live.
All in one place
Practices, with a verdict
| Practice | Why | Verdict |
|---|---|---|
| Plan understanding as tasks, not just code | Code is the smaller share of a first build | Do this |
| Discovery with a written go / no-go | Cheapest point to learn a source is broken | Do this |
| Single-number estimate on a slide | Becomes a promise before anything is known | Avoid |
| Migrate everything | Rebuilds unused reports at full cost | Avoid |
| Mock-ups without definitions and owners | Moves the argument to go-live | Avoid |
| Named source owner with an answer time | The one dependency you cannot work around | Do this |
| Access requested on day one | Approvals take weeks; start the clock early | Do this |
| Engineer as chaser | Burns builder time; answers live in one head | Avoid |
| Reconciliation target agreed and signed | Defines done before the arguments start | Do this |
| Status by percent complete per layer | Hides that nothing works yet | Avoid |
| Thin slice end to end first | Shows real progress and real problems early | Do this |
| Big-bang switch-off | No way back after the first month-end | Avoid |
| Run plan before go-live | Running is most of the life of a pipeline | Do this |
| Success as decisions changed | What the business paid for | Do this |
What good looks like
A healthier plan shape
Instead of one long build ending in a go-live date, plan five steps with a signed gate between each. The thin slice is the key: one source, all the way to a number a business owner has checked. It shows real progress early and surfaces the real problems while they are still cheap.
Discovery, thin slice, prove, widen, run. Each gate has a named person who signs it.
Watch weekly
Early warning signs
A project manager can check these every week in ten minutes. Any one of them is worth raising the same day.
- An open question to a source owner has waited more than three working days.
- An access request is still pending after the build has started.
- Status is reported in percent, and no source works end to end yet.
- A number on a report mock-up has no written definition or no owner.
- Nobody can say which report the new numbers must match, or within what tolerance.
- The same engineer is in most of the meetings this week.
- The go-live date is fixed but the parallel run is not in the plan.
- Nobody is named for on-call, reruns or the cloud bill after go-live.
Your turn
Five questions for your own project
- Where in your plan is the time to understand each source, and who answers those questions?
- Who signed the go / no-go after discovery, and what would have stopped the project?
- Which report must the new numbers match, within what tolerance, and who signs it off?
- Can you name one source that works end to end today, numbers checked?
- Which two or three business decisions should this data change, and how will you check?
FAQ
Questions people ask
Why do data projects run late more often than other projects?
Because a large part of the work is finding out what the source data means, and that depends on people outside the team. Plans written as lists of coding tasks leave that work out, so they are late from the start.
Should data projects use Agile or Waterfall?
Neither label fixes the problem. What helps is a short discovery with a go / no-go decision, then thin slices that each take one source all the way to a trusted number, with written sign-off gates.
What should a data project status report show?
What works end to end, with numbers checked, plus the open questions and how long each has waited. Percent complete per layer hides that nothing usable exists yet.
When is a data migration done?
When the agreed reports match the old ones within the agreed tolerance through at least one full business cycle, the business owner has signed, and the old system can be switched off.
Who should chase source owners and business owners?
The project manager or a technical anchor. Engineers should build from written answers, not spend their week in meetings collecting them.
Related
Where it connects
How we work · Estimation mistakes in data projects · Critical thinking in data projects · Where data engineering time goes · The technical anchor · Project review