Home › How we work › Project management mistakes

How we work · free guide · one page

Top project management mistakes in data projects, and how to avoid them

Data projects rarely fail because the code was hard. They fail because the plan treated them like ordinary software projects: a list of things to build, a date, and percent complete each week. This page shows what is different about data projects, the thirteen mistakes we see most, and what to do instead.

Teach first

Why data projects break normal project management

Three things make a data project different from building an app.

  1. It is discovery as much as delivery. You don't know what the source data really means until you look, ask and test. Some of what you learn changes the plan.
  2. The biggest dependency is not yours. The source systems, and the people who know them, belong to other teams with their own priorities.
  3. Done means trusted numbers, not shipped code. A pipeline that runs but disagrees with finance is not done. Proving the numbers is a phase of its own.
A software projectA data projectrequirementsbuildfeaturesshipthe team controls most inputsdone = the feature workslearn thesourcesbuildprove thenumbersnumbers don't match: back to learningsource system (not yours)done = people trust the numbers

A software project is mostly a straight line. A data project loops back each time the numbers don't match, and depends on systems it doesn't own.

In one line: plan the learning, get the outside people committed, prove the numbers in thin slices, and plan the running before go-live.

Mistakes: planning

Planning

1Planning as if the code were the work

What it looks like: The plan lists ingest, transform, test, deploy, each with days of coding.

Why it hurts: Most hours go to understanding what the source data means, reconciling and waiting. A plan of code tasks is late from week one.

How to avoid it: Plan the understanding work as real tasks: questions per source, answers signed, reconciliation. See where data engineering time goes.

2No discovery phase and no go / no-go gate

What it looks like: The team starts building in week one because the deadline is fixed.

Why it hurts: The project learns in month four that a key source has no change date, or that two systems disagree on the customer. The plan breaks late, when it costs most.

How to avoid it: A short discovery with a written go / no-go decision before the build is funded. See discovery: fail fast for the whole project.

3Estimates given as one number

What it looks like: "Twelve weeks." It goes on a slide and becomes a promise.

Why it hurts: Unknown sources make a single number a guess. When it slips, trust in the whole team drops.

How to avoid it: Give ranges, say what would move them, and narrow them after discovery. See estimation mistakes.

Mistakes: scope and requirements

Scope and requirements

4Scope as "migrate everything"

What it looks like: The old warehouse has 900 tables and reports, so the new one must have all of them.

Why it hurts: Half of them nobody uses. The team spends months rebuilding reports that get switched off anyway.

How to avoid it: List who uses what (query logs help), then move by decision: the reports people act on first. Retire the rest on purpose.

5Requirements as report mock-ups

What it looks like: A picture of the dashboard, with no definition behind each number.

Why it hurts: "Active customer" means three things in three departments. The engineer picks one, and the argument happens at go-live.

How to avoid it: Each number on the mock-up gets a written definition, a worked example and a named owner who signs it.

Mistakes: people and dependencies

People and dependencies

6Source owners not committed

What it looks like: The source team was told about the project, but answering questions is not in their plan.

Why it hurts: Every question waits days. The source is the one dependency the project cannot work around.

How to avoid it: Get a named source owner and an agreed answer time before the build starts. Put it in their plan, not just yours.

7Access requested when coding starts

What it looks like: Week three: the engineer asks for credentials, a firewall rule and a security review.

Why it hurts: Approvals take weeks. The team waits, or works on samples that turn out different from the real data.

How to avoid it: Raise every access request on day one, in parallel. Make "access granted" a gate before the build.

8The engineer as the stakeholder chaser

What it looks like: The engineer runs the meetings, chases answers and writes the status.

Why it hurts: Your most expensive builder spends the week in calls. Answers stay in their head and leave with them.

How to avoid it: The PM or a technical anchor chases and writes answers down; the engineer builds from what is written.

what happensengineersource ownerbusiness ownersecurityreport userwhat worksPM or anchorsource ownersecuritybusiness ownerengineerbuilds fromwritten answers

The chasing has to happen. It should not land on the person who builds.

Mistakes: delivery and quality

Delivery and quality

9No reconciliation target or sign-off

What it looks like: "It should match the old report", with no tolerance and no one to sign.

Why it hurts: Every small difference becomes an investigation. Go-live keeps moving because "done" was never defined.

How to avoid it: Agree up front: which report, which tolerance, the known differences, and who signs off. Automate the check.

10Status by percent complete

What it looks like: "Bronze 80%, silver 60%, gold 30%: 54% done."

Why it hurts: Nothing works for a user until the last layer is done, so the percentages hide the real risk.

How to avoid it: Report what works end to end: "orders reach the report and match finance". Build thin slices so that sentence comes early.

11Big-bang go-live with no parallel run

What it looks like: The old system is switched off the day the new one goes live.

Why it hurts: The first month-end close is the first real test, and there is no way back.

How to avoid it: Run old and new side by side for at least one full business cycle, such as a month-end, and compare daily.

"54% done""orders work end to end"ingest90%bronze80%silver60%gold30%report10%busy everywhere, nothing a user can useingestbronzesilvergoldreportone source: ordersmatches finance within 0.5%one thing works, and everyone can see it

Percent complete per layer looks like progress. One source working end to end is progress.

Mistakes: after go-live

After go-live

12No plan for running it

What it looks like: The project ends at go-live. Nobody owns failures, reruns, source changes or the cloud bill.

Why it hurts: Quality fixes and reruns become the biggest share of the work, done by people already moved to the next project.

How to avoid it: Plan the run before go-live: who is on call, how reruns and backfills work, alerts on source changes, a cost budget.

13Success measured as pipelines built

What it looks like: The closing slide counts tables, pipelines and terabytes.

Why it hurts: Nobody checks whether anyone decided anything differently. The next budget is hard to win.

How to avoid it: Name two or three decisions the data should change, at the start. Check them three months after go-live.

All in one place

Practices, with a verdict

PracticeWhyVerdict
Plan understanding as tasks, not just codeCode is the smaller share of a first buildDo this
Discovery with a written go / no-goCheapest point to learn a source is brokenDo this
Single-number estimate on a slideBecomes a promise before anything is knownAvoid
Migrate everythingRebuilds unused reports at full costAvoid
Mock-ups without definitions and ownersMoves the argument to go-liveAvoid
Named source owner with an answer timeThe one dependency you cannot work aroundDo this
Access requested on day oneApprovals take weeks; start the clock earlyDo this
Engineer as chaserBurns builder time; answers live in one headAvoid
Reconciliation target agreed and signedDefines done before the arguments startDo this
Status by percent complete per layerHides that nothing works yetAvoid
Thin slice end to end firstShows real progress and real problems earlyDo this
Big-bang switch-offNo way back after the first month-endAvoid
Run plan before go-liveRunning is most of the life of a pipelineDo this
Success as decisions changedWhat the business paid forDo this

What good looks like

A healthier plan shape

Instead of one long build ending in a go-live date, plan five steps with a signed gate between each. The thin slice is the key: one source, all the way to a number a business owner has checked. It shows real progress early and surfaces the real problems while they are still cheap.

discovery2-4 weeksgate: go / no-gothin sliceone source,end to endgate: slice worksprove numbersreconcile,parallel rungate: numbers signedwidennext sources,one at a timegate: handoverrunsupport, reruns,costeach gate has a named person who signs it, in writing

Discovery, thin slice, prove, widen, run. Each gate has a named person who signs it.

Watch weekly

Early warning signs

A project manager can check these every week in ten minutes. Any one of them is worth raising the same day.

  • An open question to a source owner has waited more than three working days.
  • An access request is still pending after the build has started.
  • Status is reported in percent, and no source works end to end yet.
  • A number on a report mock-up has no written definition or no owner.
  • Nobody can say which report the new numbers must match, or within what tolerance.
  • The same engineer is in most of the meetings this week.
  • The go-live date is fixed but the parallel run is not in the plan.
  • Nobody is named for on-call, reruns or the cloud bill after go-live.

Your turn

Five questions for your own project

  1. Where in your plan is the time to understand each source, and who answers those questions?
  2. Who signed the go / no-go after discovery, and what would have stopped the project?
  3. Which report must the new numbers match, within what tolerance, and who signs it off?
  4. Can you name one source that works end to end today, numbers checked?
  5. Which two or three business decisions should this data change, and how will you check?

FAQ

Questions people ask

Why do data projects run late more often than other projects?

Because a large part of the work is finding out what the source data means, and that depends on people outside the team. Plans written as lists of coding tasks leave that work out, so they are late from the start.

Should data projects use Agile or Waterfall?

Neither label fixes the problem. What helps is a short discovery with a go / no-go decision, then thin slices that each take one source all the way to a trusted number, with written sign-off gates.

What should a data project status report show?

What works end to end, with numbers checked, plus the open questions and how long each has waited. Percent complete per layer hides that nothing usable exists yet.

When is a data migration done?

When the agreed reports match the old ones within the agreed tolerance through at least one full business cycle, the business owner has signed, and the old system can be switched off.

Who should chase source owners and business owners?

The project manager or a technical anchor. Engineers should build from written answers, not spend their week in meetings collecting them.

Related

Where it connects

How we work · Estimation mistakes in data projects · Critical thinking in data projects · Where data engineering time goes · The technical anchor · Project review