Home › Learn › The technical anchor

Data engineering · a guide for developers stepping up

The technical anchor: how to think for a whole data project

On most data projects, one developer ends up holding the whole technical picture: how the sources, the pipelines, the platform, the tests, the release and the people all fit together. Call that person the technical anchor. This guide is about how to think in that seat: proactive, big picture, and failing fast on design, estimates, scope and decisions, for yourself and for everyone around you.

Who is the technical anchor?

What a technical anchor is

Not the boss, and not only the architect. The technical anchor is the person the whole project can lean on for one question: will this actually work, end to end, for everyone who depends on it? Sometimes the title is tech lead, lead engineer or delivery architect. The title matters less than the habit.

An anchor holds a ship steady while everything around it moves. On a project, requirements shift, sources change, dates slip and people come and go. The anchor keeps the technical picture steady and connected, so the project doesn't drift.

The anchor still writes code. But their most valuable work is often not code at all: a question asked in week one, a decision written down, an access request sent early, a risk raised before it bites.

Why does one role matter so much?

Why one role decides success

BusinessAnalysisDesignBuildPlatformTestRuna definitionnobody signedan edge casenot writtena decisionnot recordedaccess askedtoo latetests writtenafter the buildno runbookfor night oneeach team does its own part well; the project fails in the gaps between themthe technical anchor's job is to watch the gaps
Projects rarely fail inside a team. They fail between teams, where nobody owns the hand-off.

Look at how data projects actually fail. Rarely because a developer couldn't write the code. Usually because of something between teams: a definition of revenue nobody signed, a source that deletes records without telling anyone, a firewall rule requested too late, tests written after the build, an operations team that met the pipeline on go-live night.

Each team did its own part. Nobody owned the seams. That is the technical anchor's territory.

Who do I need to think about?

Everyone around you: the map

The anchor sits in the middle of at least twelve roles. Each one needs something from you, and you need something from each of them. For every role there is also a question that helps them find their own mistakes early, which is where most of the anchor's value comes from.

Technical anchorholds the whole pictureBusiness owner, usersBusiness analystArchitectPM and Agile leadDevelopersPlatform teamSource system ownersData consumersTest and QARelease and DevOpsOperations and supportSecurity, governance

Click any role (or tab to it and press Enter).

On a small project one person may wear several of these hats. The roles still exist; someone has to think about each one.

What is actually on my plate?

What the anchor owns

You won't do all of this yourself. But you own making sure it happens, and noticing when it isn't.

AreaWhat "done well" looks like
Design and architectureOne end-to-end design that works from every angle, written as short decision records
ScopeA written in-scope and out-of-scope list, signed, with a change process
EstimationRanges with assumptions, corrected by real progress
Requirements into checksEvery requirement turned into something the data can be tested against
Decisions and sign-offA decision log: options, choice, reason, owner, who signed, when
Risks and dependenciesTop risks with an owner, a trigger and a mitigation, reviewed weekly
QualityA test strategy from day one: data checks, reconciliation, performance
Release and runDeployment working early, cut-over rehearsed, operations ready before go-live
The teamStandards, reviews, unblocking; making the next person able to do this job
CommunicationBad news early, in plain words, with a next step

What do I do at each stage of the project?

The data project lifecycle, stage by stage

Software teams talk about the SDLC, the software development lifecycle. A data project has its own version, which this page calls the data development lifecycle (DDLC). The stages look familiar, but the risks are different: the data itself is the biggest unknown, so the anchor's job at every stage is to look at real data sooner than anyone expects.

1 Requirement received2 Rough estimate3 Discovery4 Discovery sign-offsign-off gate5 MVP 1 starts6 Build7 Test8 Deploy9 Production sign-offsign-off gate10 Project sign-offsign-off gatedeciding: is it worth doing, and how?delivering: one MVP at a timestages 5 to 9 repeat for every MVP
Ten stages, three sign-off gates (gold). The first four decide whether and how; the rest deliver one MVP at a time.

1 · Requirement received

A business need arrives: an email, a ticket, a one-page brief.

The anchor: Read it for what is missing. Ask the first five questions: who will use it, which decisions it drives, which sources feed it, how fresh it must be, and what exists today. Write down every assumption you are making.

Fail-fast question: What would make this impossible, or three times bigger?

Hands over: A list of open questions and assumptions.

2 · Rough estimate

Someone asks "roughly how big is this?" before anyone has looked at the data.

The anchor: Count the things you can see (sources, tables, reports, interfaces) and give a range, not a number, with its assumptions. Say plainly that it can move by half either way until discovery. Estimate the discovery itself too.

Fail-fast question: Which assumption, if wrong, moves this estimate the most?

Hands over: An estimate range, its assumptions, and the cost of discovery.

3 · Discovery

A short, timeboxed stage to find out whether and how the project can work.

The anchor: Profile real data, build a thin slice or spikes for the riskiest beliefs, shape the architecture, walk the scope traps, estimate per thing, and draft the test strategy, with the project manager and business analyst covering their parts. Discovery, step by step →

Fail-fast question: Which belief, if wrong, would sink the whole project, and have we tested it yet?

Hands over: The discovery document.

4 · Discovery sign-off sign-off

The sponsor decides: go, go with changes, or stop.

The anchor: Walk the reviewers through the risks honestly, including the bad news. Get scope, architecture, estimate and the key decisions signed by the people who own them.

Fail-fast question: Is anyone signing something they haven't read or don't agree with?

Hands over: A signed discovery document, a baseline estimate, and the backlog for MVP 1.

5 · MVP 1 starts

The first minimum viable product: the smallest slice that gives real value.

The anchor: Get environments, access, the release pipeline and coding standards ready in the first days. Make sure every backlog item has acceptance criteria and data tests, and that "done" includes tested and deployed.

Fail-fast question: Can we deploy the thin slice to the test environment this week?

Hands over: A team that can build, test and deploy from day one.

6 · Build

Developers build the pipelines, models and reports.

The anchor: Review designs and code, keep standards, unblock people, track actual effort against the estimate every week, demo real numbers every two weeks, and push every scope change through the agreed change process.

Fail-fast question: Which piece of work is turning out harder than it looked?

Hands over: Working increments, demoed with real numbers.

7 · Test

The system is checked against the requirements and the real data.

The anchor: Make sure numbers reconcile from source to report, data quality rules run, performance holds at real volume, a rerun doesn't double anything, and business users test with numbers they recognise.

Fail-fast question: Would the business recognise these numbers as right?

Hands over: Test evidence and user acceptance sign-off.

8 · Deploy

The release goes to production.

The anchor: Rehearse the cut-over and the rollback, plan the history load, switch on monitoring and alerts, finish the runbook, and run a failure drill with operations before go-live.

Fail-fast question: Have we rehearsed both the cut-over and the rollback?

Hands over: A production release and a runbook operations can use.

9 · Production sign-off sign-off

The business and operations accept the system in production.

The anchor: Watch the first production runs closely (often called hypercare), reconcile the first loads, close critical defects, and confirm operations can run it alone. Then plan the next MVP: stages 5 to 9 repeat for each one.

Fail-fast question: Would operations accept this if the team left tomorrow?

Hands over: Production acceptance, and the plan for the next MVP.

10 · Project sign-off sign-off

All MVPs are live and the project closes.

The anchor: Hand over the decision log, runbook, data dictionary and tests. Have the support team make one real change while you watch. Close the risk log, measure the benefits promised at the start, and write down what you learned.

Fail-fast question: Can the team run and change this without us?

Hands over: Project closure, hand-over and lessons learned.

Names vary between companies (inception, mobilisation, hypercare, go-live). The jobs at each stage are the same.

How should I think, day to day?

The mindset: proactive, big picture, fail fast

timeDiscoverDesignBuildTestDeployRuncost of a changeProfile real dataweek 1Thin slice, end to endweeks 1 to 2Re-estimateafter the sliceSign scope and decisionsbefore build
The anchor pulls every discovery to the left of the curve, where changing course is still cheap.

How do I fail fast on the architecture?

Fail fast on architecture

  1. List the design's riskiest beliefs. "CDC from the source is possible." "The volume fits a daily batch." "The gold table can serve the dashboard at this grain." "The platform allows this pattern."
  2. Build a thin slice, end to end, in the first two weeks. One source table, through landing, bronze, silver and gold, to one dashboard tile or one query, deployed to a production-like environment. It will expose access, network, format, volume and tooling problems that no diagram will. The data platform architecture guide shows the layers it should cross.
  3. Spike the unknowns, with a timebox. "Two days to prove we can read the mainframe extract. If not, we escalate." A spike without a timebox is just slow building.
  4. Write short decision records. One page each: the question, the options, what you chose, why, and what would make you revisit it. They let others challenge the design early, and stop it being re-argued later.
  5. Review it with the whole map, not only architects. Ask operations how they would rerun it, security where personal data flows, testers how they would check it, consumers whether the grain fits.
  6. Know your kill line. Decide in advance what result would make you change the design, before the spike runs.

What makes a design good for this project?

A design that works from every side

A design usually reflects the angle its author cares about most: speed, elegance, cost, or the tool they know. A good design for a project works from every side. Look at it through eight lenses, and write one answer per lens.

Businessright question,right definitionsDatagrain, history,deletes, late rowsSecuritywho sees what,personal dataCostper run, per month,at 10x volumePerformancevolume todayand in two yearsOperationsrerun, monitor,recoverChangenew source, newfield, new consumerTestabilitycan each rulebe checked?Your designseen 8 ways
A design is good when it survives every lens, not just the one its author cares about most.

For data projects, the Data lens hides the most traps. Ask early: what is the grain of each table; does the source delete or correct records; can rows arrive late; do we need history; can the same file arrive twice? Each of these is a recorded scene in the Problem Lab, with what goes wrong and the right fix. Concepts worth knowing cold: idempotency, incremental loads, slowly changing dimensions, schema evolution and data contracts.

How do I stop estimates going wrong?

Fail fast on estimates

1 Count the thingssources, tables, rules,reports, interfaces2 Build one, measurethe thin slice givesyour real rate3 Range, not a pointbest, likely, worst,with assumptions4 Re-check weeklyplanned vs actual;say the gap earlyUnknowns get a timeboxed spike, not a guessan estimate is a belief too: test it the same way
Estimates fail fast when the first real piece of work is used to correct them.

What is in, what is out?

In scope, out of scope

Anything not written as out of scope will be assumed to be in. Data projects have the same ten traps every time. Put each one explicitly in or out, in writing, and get it signed.

Scope trapThe question to settle
History and backfillDo we load five years of history, or start from today? Who checks the old numbers?
Deletes at the sourceWhen a record is deleted upstream, must it disappear here, be flagged, or stay?
Late and corrected dataIf yesterday's sales are corrected next week, which days do we rebuild?
Changes over timeDo we keep history of a customer's address (slowly changing dimensions) or only the latest?
Bad rowsReject the batch, quarantine the rows, or load and flag them? Who fixes them?
Data quality at the sourceAre we expected to fix the source's mistakes, or report them?
Personal dataMasking, retention, the right to be forgotten: designed in, or someone else's job?
Reports and dashboardsAre the reports in scope, or only the tables they read?
Access and onboardingWho requests access for users, and who approves it?
RecoveryHow fast must it recover after a failure, and how much data can be lost?

Keep a "parking lot" for good ideas that are out of scope, so saying no doesn't feel like losing them. Any change after sign-off goes through the agreed change process, with its effect on the estimate shown. The concept page on scoping out has more.

Which decisions need a signature?

Decisions and sign-off

One missed requirement or one decision nobody signed can cost weeks at the end. A decision that isn't written down will be reopened by someone who wasn't in the room. Keep a decision log from day one.

Decision:    What counts as revenue in the sales dashboard
Options:     A gross incl. tax   B net of tax   C net of tax and returns
Chosen:      C
Why:         matches the finance month-end report
Owner:       Head of sales analytics
Signed by:   Finance controller, 12 Nov
Revisit if:  finance changes its month-end definition

The decisions data projects most often leave unsigned:

KindExample
DefinitionsWhat counts as revenue: gross or net, with or without tax, booked or paid?
Source of truthWhen the CRM and the billing system disagree on a customer, which one wins?
FreshnessDaily by 7 a.m., hourly, or near real time? Each costs very differently.
HistoryWhich tables keep history, and for how long?
Bad dataWhat happens to a row that fails a rule, and who is told?
Tools and patternsBatch or streaming; which ingestion pattern; build or reuse a framework.

Give each open decision a deadline and a default: "If we don't hear by Friday, we go with option C." It keeps the project moving and makes the cost of delay visible.

How do I manage stakeholders, beyond my own manager?

Help your stakeholders fail fast

Managing stakeholders is not only keeping them informed. The bigger job is helping each of them find their mistakes early: a business definition that doesn't match the data, a security rule nobody mentioned, an operations need nobody asked about. If they find it at go-live, it is your project that fails.

Click through the roles in the map above: each one has its own fail-fast question.

How do I keep risks under control?

Risks and dependencies

Keep one short log with four kinds of item, often called RAID: Risks (might happen), Assumptions (believed, not yet proved), Issues (happening now) and Dependencies (waiting on someone). Keep the top five visible and review them every week.

Risk:        Source team release on 20 Nov may rename columns
Likelihood:  medium     Impact: high (silver loads fail)
Trigger:     release notes not shared by 13 Nov
Mitigation:  contract check before each write; mapping in the manifest
Owner:       Technical anchor, with source system owner
Review:      weekly

A risk without an owner and a trigger is a worry, not a risk. Assumptions are the most dangerous entries, because they look settled: test them like any other belief.

How do I avoid a painful go-live?

Test, deploy and run, from week one

I've just been made the technical anchor. Where do I start?

Your first 30 days

WhenWhat
Days 1 to 3Meet every role on the map. Ask each: what does success look like for you, and what has gone wrong before? Request all access and environments now.
Week 1Pull and profile real data from every source. Write the first list of beliefs the plan depends on. Start the risk and decision logs.
Weeks 1 to 2Build a thin slice: one source table, through every layer, to one tile or one query, deployed to a production-like environment.
Week 2Show real numbers to the business. Write the in-scope and out-of-scope list. Take the first decisions to sign-off.
Weeks 3 to 4Re-estimate from the slice. Agree the test strategy and the first data contract. Review the design through every lens with the people on the map, not only architects.

What does a normal week look like?

A weekly rhythm

WhenWhat
MondayTop five risks: any moved? Decisions due this week, and who must sign.
MidweekWalk the board with developers: anything harder than it looked? Unblock or escalate.
Every two weeksDemo with real numbers to the business. Re-estimate. Update scope if anything changed, in writing.
FridayThree lines to stakeholders: what we learned, what changed, what we need from you.

What do new anchors get wrong?

Mistakes

  1. Being the best developer instead of the anchor. Writing the hardest code yourself while the gaps between teams go unwatched.
  2. Designing on paper for weeks. A beautiful design that has never touched real data or a real environment.
  3. A single-number estimate. 'Six weeks' becomes a promise. 'Five to eight weeks, if these three things hold' is a plan.
  4. Verbal decisions. Agreed in a meeting, never written down, reopened a month later by someone who wasn't there.
  5. Silent scope. Anything not written as out of scope will be assumed to be in.
  6. Only managing upward. Keeping the project manager informed while source owners, operations and security hear nothing until they are blocking.
  7. Green until it's red. Hiding a slipping date until it can't be fixed. Bad news early is a gift.

Self-check

Questions to check yourself

  1. Can you draw the whole system, from every source to every consumer, on one page?
  2. Which three beliefs does the plan depend on most, and has each been tested against real data?
  3. Is there a written list of what is out of scope, and has the business owner signed it?
  4. Which open decisions have a deadline and an owner?
  5. Has anything been deployed to a production-like environment yet?
  6. Do operations know what to do if the pipeline fails tonight?
  7. Which stakeholder have you not spoken to in two weeks?
  8. What is your estimate range today, and how did it change since the thin slice?

The anchor doesn't make every decision or write every line. The anchor makes sure nothing important falls between the people who do.

Next

Where to go next

Check your own project: the project review skill reads your documents and says what is missing at your stage.

The thinking habit underneath: fail fast, own the result. The customer-facing cousin of this role: what a forward-deployed engineer does.

The system you are anchoring: data platform architecture, the medallion, and 19 data problems, recorded. The path: data engineering, crawl to run.