Home › Learn › The technical anchor
Data engineering · a guide for developers stepping up
The technical anchor: how to think for a whole data project
On most data projects, one developer ends up holding the whole technical picture: how the sources, the pipelines, the platform, the tests, the release and the people all fit together. Call that person the technical anchor. This guide is about how to think in that seat: proactive, big picture, and failing fast on design, estimates, scope and decisions, for yourself and for everyone around you.
Who is the technical anchor?
What a technical anchor is
Not the boss, and not only the architect. The technical anchor is the person the whole project can lean on for one question: will this actually work, end to end, for everyone who depends on it? Sometimes the title is tech lead, lead engineer or delivery architect. The title matters less than the habit.
An anchor holds a ship steady while everything around it moves. On a project, requirements shift, sources change, dates slip and people come and go. The anchor keeps the technical picture steady and connected, so the project doesn't drift.
The anchor still writes code. But their most valuable work is often not code at all: a question asked in week one, a decision written down, an access request sent early, a risk raised before it bites.
Why does one role matter so much?
Why one role decides success
Look at how data projects actually fail. Rarely because a developer couldn't write the code. Usually because of something between teams: a definition of revenue nobody signed, a source that deletes records without telling anyone, a firewall rule requested too late, tests written after the build, an operations team that met the pipeline on go-live night.
Each team did its own part. Nobody owned the seams. That is the technical anchor's territory.
Who do I need to think about?
Everyone around you: the map
The anchor sits in the middle of at least twelve roles. Each one needs something from you, and you need something from each of them. For every role there is also a question that helps them find their own mistakes early, which is where most of the anchor's value comes from.
Click any role (or tab to it and press Enter).
On a small project one person may wear several of these hats. The roles still exist; someone has to think about each one.
What is actually on my plate?
What the anchor owns
You won't do all of this yourself. But you own making sure it happens, and noticing when it isn't.
| Area | What "done well" looks like |
|---|---|
| Design and architecture | One end-to-end design that works from every angle, written as short decision records |
| Scope | A written in-scope and out-of-scope list, signed, with a change process |
| Estimation | Ranges with assumptions, corrected by real progress |
| Requirements into checks | Every requirement turned into something the data can be tested against |
| Decisions and sign-off | A decision log: options, choice, reason, owner, who signed, when |
| Risks and dependencies | Top risks with an owner, a trigger and a mitigation, reviewed weekly |
| Quality | A test strategy from day one: data checks, reconciliation, performance |
| Release and run | Deployment working early, cut-over rehearsed, operations ready before go-live |
| The team | Standards, reviews, unblocking; making the next person able to do this job |
| Communication | Bad news early, in plain words, with a next step |
What do I do at each stage of the project?
The data project lifecycle, stage by stage
Software teams talk about the SDLC, the software development lifecycle. A data project has its own version, which this page calls the data development lifecycle (DDLC). The stages look familiar, but the risks are different: the data itself is the biggest unknown, so the anchor's job at every stage is to look at real data sooner than anyone expects.
1 · Requirement received
A business need arrives: an email, a ticket, a one-page brief.
The anchor: Read it for what is missing. Ask the first five questions: who will use it, which decisions it drives, which sources feed it, how fresh it must be, and what exists today. Write down every assumption you are making.
Fail-fast question: What would make this impossible, or three times bigger?
Hands over: A list of open questions and assumptions.
2 · Rough estimate
Someone asks "roughly how big is this?" before anyone has looked at the data.
The anchor: Count the things you can see (sources, tables, reports, interfaces) and give a range, not a number, with its assumptions. Say plainly that it can move by half either way until discovery. Estimate the discovery itself too.
Fail-fast question: Which assumption, if wrong, moves this estimate the most?
Hands over: An estimate range, its assumptions, and the cost of discovery.
3 · Discovery
A short, timeboxed stage to find out whether and how the project can work.
The anchor: Profile real data, build a thin slice or spikes for the riskiest beliefs, shape the architecture, walk the scope traps, estimate per thing, and draft the test strategy, with the project manager and business analyst covering their parts. Discovery, step by step →
Fail-fast question: Which belief, if wrong, would sink the whole project, and have we tested it yet?
Hands over: The discovery document.
4 · Discovery sign-off sign-off
The sponsor decides: go, go with changes, or stop.
The anchor: Walk the reviewers through the risks honestly, including the bad news. Get scope, architecture, estimate and the key decisions signed by the people who own them.
Fail-fast question: Is anyone signing something they haven't read or don't agree with?
Hands over: A signed discovery document, a baseline estimate, and the backlog for MVP 1.
5 · MVP 1 starts
The first minimum viable product: the smallest slice that gives real value.
The anchor: Get environments, access, the release pipeline and coding standards ready in the first days. Make sure every backlog item has acceptance criteria and data tests, and that "done" includes tested and deployed.
Fail-fast question: Can we deploy the thin slice to the test environment this week?
Hands over: A team that can build, test and deploy from day one.
6 · Build
Developers build the pipelines, models and reports.
The anchor: Review designs and code, keep standards, unblock people, track actual effort against the estimate every week, demo real numbers every two weeks, and push every scope change through the agreed change process.
Fail-fast question: Which piece of work is turning out harder than it looked?
Hands over: Working increments, demoed with real numbers.
7 · Test
The system is checked against the requirements and the real data.
The anchor: Make sure numbers reconcile from source to report, data quality rules run, performance holds at real volume, a rerun doesn't double anything, and business users test with numbers they recognise.
Fail-fast question: Would the business recognise these numbers as right?
Hands over: Test evidence and user acceptance sign-off.
8 · Deploy
The release goes to production.
The anchor: Rehearse the cut-over and the rollback, plan the history load, switch on monitoring and alerts, finish the runbook, and run a failure drill with operations before go-live.
Fail-fast question: Have we rehearsed both the cut-over and the rollback?
Hands over: A production release and a runbook operations can use.
9 · Production sign-off sign-off
The business and operations accept the system in production.
The anchor: Watch the first production runs closely (often called hypercare), reconcile the first loads, close critical defects, and confirm operations can run it alone. Then plan the next MVP: stages 5 to 9 repeat for each one.
Fail-fast question: Would operations accept this if the team left tomorrow?
Hands over: Production acceptance, and the plan for the next MVP.
10 · Project sign-off sign-off
All MVPs are live and the project closes.
The anchor: Hand over the decision log, runbook, data dictionary and tests. Have the support team make one real change while you watch. Close the risk log, measure the benefits promised at the start, and write down what you learned.
Fail-fast question: Can the team run and change this without us?
Hands over: Project closure, hand-over and lessons learned.
Names vary between companies (inception, mobilisation, hypercare, go-live). The jobs at each stage are the same.
How should I think, day to day?
The mindset: proactive, big picture, fail fast
- Proactive. Don't wait to be asked. If a dependency will take six weeks, the right day to request it is today. If a stakeholder hasn't seen real numbers, show them before they ask.
- Big picture. Every change has an upstream and a downstream. Before changing a table, ask who feeds it and who reads it. Before choosing a pattern, ask which company goal the project serves, and whether this choice helps it.
- Fail fast. Every plan stands on beliefs: the data exists, the source can take the load, the business agrees on the definitions, the estimate holds. Write them down, and test the riskiest ones first, in the cheapest way that could prove you wrong. The blog post fail fast, own the result walks through the method.
How do I fail fast on the architecture?
Fail fast on architecture
- List the design's riskiest beliefs. "CDC from the source is possible." "The volume fits a daily batch." "The gold table can serve the dashboard at this grain." "The platform allows this pattern."
- Build a thin slice, end to end, in the first two weeks. One source table, through landing, bronze, silver and gold, to one dashboard tile or one query, deployed to a production-like environment. It will expose access, network, format, volume and tooling problems that no diagram will. The data platform architecture guide shows the layers it should cross.
- Spike the unknowns, with a timebox. "Two days to prove we can read the mainframe extract. If not, we escalate." A spike without a timebox is just slow building.
- Write short decision records. One page each: the question, the options, what you chose, why, and what would make you revisit it. They let others challenge the design early, and stop it being re-argued later.
- Review it with the whole map, not only architects. Ask operations how they would rerun it, security where personal data flows, testers how they would check it, consumers whether the grain fits.
- Know your kill line. Decide in advance what result would make you change the design, before the spike runs.
What makes a design good for this project?
A design that works from every side
A design usually reflects the angle its author cares about most: speed, elegance, cost, or the tool they know. A good design for a project works from every side. Look at it through eight lenses, and write one answer per lens.
For data projects, the Data lens hides the most traps. Ask early: what is the grain of each table; does the source delete or correct records; can rows arrive late; do we need history; can the same file arrive twice? Each of these is a recorded scene in the Problem Lab, with what goes wrong and the right fix. Concepts worth knowing cold: idempotency, incremental loads, slowly changing dimensions, schema evolution and data contracts.
How do I stop estimates going wrong?
Fail fast on estimates
- Count before you guess. Sources, tables, transformation rules, reports, interfaces, environments. Data projects are lists of similar things; estimate per thing.
- Let the thin slice correct you. Say the first table took six days including setting up the framework, and the next three took one and a half days each. Now you have a real rate for the remaining forty. (These numbers are an example.)
- Give a range with its assumptions. "Seven to ten weeks, assuming source access by the 15th, no history load, and definitions signed by week three." When an assumption breaks, the estimate changes, visibly and without blame.
- Estimate the invisible work. Access, environments, testing, data fixes, performance tuning, documentation, hand-over, the cut-over weekend. It is usually a third or more of the effort, and usually left out.
- Track planned against actual every week, and say the gap the week it appears.
What is in, what is out?
In scope, out of scope
Anything not written as out of scope will be assumed to be in. Data projects have the same ten traps every time. Put each one explicitly in or out, in writing, and get it signed.
| Scope trap | The question to settle |
|---|---|
| History and backfill | Do we load five years of history, or start from today? Who checks the old numbers? |
| Deletes at the source | When a record is deleted upstream, must it disappear here, be flagged, or stay? |
| Late and corrected data | If yesterday's sales are corrected next week, which days do we rebuild? |
| Changes over time | Do we keep history of a customer's address (slowly changing dimensions) or only the latest? |
| Bad rows | Reject the batch, quarantine the rows, or load and flag them? Who fixes them? |
| Data quality at the source | Are we expected to fix the source's mistakes, or report them? |
| Personal data | Masking, retention, the right to be forgotten: designed in, or someone else's job? |
| Reports and dashboards | Are the reports in scope, or only the tables they read? |
| Access and onboarding | Who requests access for users, and who approves it? |
| Recovery | How fast must it recover after a failure, and how much data can be lost? |
Keep a "parking lot" for good ideas that are out of scope, so saying no doesn't feel like losing them. Any change after sign-off goes through the agreed change process, with its effect on the estimate shown. The concept page on scoping out has more.
Which decisions need a signature?
Decisions and sign-off
One missed requirement or one decision nobody signed can cost weeks at the end. A decision that isn't written down will be reopened by someone who wasn't in the room. Keep a decision log from day one.
Decision: What counts as revenue in the sales dashboard Options: A gross incl. tax B net of tax C net of tax and returns Chosen: C Why: matches the finance month-end report Owner: Head of sales analytics Signed by: Finance controller, 12 Nov Revisit if: finance changes its month-end definition
The decisions data projects most often leave unsigned:
| Kind | Example |
|---|---|
| Definitions | What counts as revenue: gross or net, with or without tax, booked or paid? |
| Source of truth | When the CRM and the billing system disagree on a customer, which one wins? |
| Freshness | Daily by 7 a.m., hourly, or near real time? Each costs very differently. |
| History | Which tables keep history, and for how long? |
| Bad data | What happens to a row that fails a rule, and who is told? |
| Tools and patterns | Batch or streaming; which ingestion pattern; build or reuse a framework. |
Give each open decision a deadline and a default: "If we don't hear by Friday, we go with option C." It keeps the project moving and makes the cost of delay visible.
How do I manage stakeholders, beyond my own manager?
Help your stakeholders fail fast
Managing stakeholders is not only keeping them informed. The bigger job is helping each of them find their mistakes early: a business definition that doesn't match the data, a security rule nobody mentioned, an operations need nobody asked about. If they find it at go-live, it is your project that fails.
- Show real data early. A profile of their own data in week one or two surfaces wrong assumptions faster than any meeting.
- Demo every two weeks with real numbers, not slides. People react to numbers they recognise.
- Ask the question they haven't thought of. "What should happen to a customer who is deleted in the CRM?" "Who reruns this if it fails on a holiday?"
- Make the cost of a late change visible. "Changing the grain now costs two days; after testing it costs three weeks."
- Go beyond your immediate stakeholder. Source owners, operations, security and downstream consumers are often not in the weekly meeting, and they are the ones who block go-live.
- Bad news early, with a next step. "The source can't provide deletes. We propose a weekly key reconciliation instead; it adds three days. We need your decision by Thursday."
Click through the roles in the map above: each one has its own fail-fast question.
How do I keep risks under control?
Risks and dependencies
Keep one short log with four kinds of item, often called RAID: Risks (might happen), Assumptions (believed, not yet proved), Issues (happening now) and Dependencies (waiting on someone). Keep the top five visible and review them every week.
Risk: Source team release on 20 Nov may rename columns Likelihood: medium Impact: high (silver loads fail) Trigger: release notes not shared by 13 Nov Mitigation: contract check before each write; mapping in the manifest Owner: Technical anchor, with source system owner Review: weekly
A risk without an owner and a trigger is a worry, not a risk. Assumptions are the most dangerous entries, because they look settled: test them like any other belief.
How do I avoid a painful go-live?
Test, deploy and run, from week one
- Test from the first pipeline. Row counts, sums and keys reconciled from source to gold; data quality rules on each table; a check that a rerun doesn't double anything. Tests written after the build find problems when they are most expensive.
- Deploy from week one. The thin slice should already travel through the release pipeline. The first production deployment should never be go-live day.
- Design for the night shift. For each pipeline, write how to tell it failed, how to rerun it safely, and who to call. Run a failure drill with operations before go-live.
- Rehearse the cut-over. Write the steps, run them in a test environment, time them, and agree the rollback point.
- Plan the hand-over. A runbook, the decision log, the tests the support team can rerun, and one person on their side who has made a change while you watched. The concept page on hand-over has the checklist.
I've just been made the technical anchor. Where do I start?
Your first 30 days
| When | What |
|---|---|
| Days 1 to 3 | Meet every role on the map. Ask each: what does success look like for you, and what has gone wrong before? Request all access and environments now. |
| Week 1 | Pull and profile real data from every source. Write the first list of beliefs the plan depends on. Start the risk and decision logs. |
| Weeks 1 to 2 | Build a thin slice: one source table, through every layer, to one tile or one query, deployed to a production-like environment. |
| Week 2 | Show real numbers to the business. Write the in-scope and out-of-scope list. Take the first decisions to sign-off. |
| Weeks 3 to 4 | Re-estimate from the slice. Agree the test strategy and the first data contract. Review the design through every lens with the people on the map, not only architects. |
What does a normal week look like?
A weekly rhythm
| When | What |
|---|---|
| Monday | Top five risks: any moved? Decisions due this week, and who must sign. |
| Midweek | Walk the board with developers: anything harder than it looked? Unblock or escalate. |
| Every two weeks | Demo with real numbers to the business. Re-estimate. Update scope if anything changed, in writing. |
| Friday | Three lines to stakeholders: what we learned, what changed, what we need from you. |
What do new anchors get wrong?
Mistakes
- Being the best developer instead of the anchor. Writing the hardest code yourself while the gaps between teams go unwatched.
- Designing on paper for weeks. A beautiful design that has never touched real data or a real environment.
- A single-number estimate. 'Six weeks' becomes a promise. 'Five to eight weeks, if these three things hold' is a plan.
- Verbal decisions. Agreed in a meeting, never written down, reopened a month later by someone who wasn't there.
- Silent scope. Anything not written as out of scope will be assumed to be in.
- Only managing upward. Keeping the project manager informed while source owners, operations and security hear nothing until they are blocking.
- Green until it's red. Hiding a slipping date until it can't be fixed. Bad news early is a gift.
Self-check
Questions to check yourself
- Can you draw the whole system, from every source to every consumer, on one page?
- Which three beliefs does the plan depend on most, and has each been tested against real data?
- Is there a written list of what is out of scope, and has the business owner signed it?
- Which open decisions have a deadline and an owner?
- Has anything been deployed to a production-like environment yet?
- Do operations know what to do if the pipeline fails tonight?
- Which stakeholder have you not spoken to in two weeks?
- What is your estimate range today, and how did it change since the thin slice?
The anchor doesn't make every decision or write every line. The anchor makes sure nothing important falls between the people who do.
Next
Where to go next
Check your own project: the project review skill reads your documents and says what is missing at your stage.
The thinking habit underneath: fail fast, own the result. The customer-facing cousin of this role: what a forward-deployed engineer does.
The system you are anchoring: data platform architecture, the medallion, and 19 data problems, recorded. The path: data engineering, crawl to run.