Home › Data engineering › Where the time goes

Data engineering · free guide · one page

Where data engineering time really goes, and how to win it back

Ask someone what a data engineer does and they say "writes pipelines". Watch one for a month and you see something else: most hours go to finding out what the source data means, chasing people, and making numbers match. This page shows where the time goes, why the meaning of source data sits at the root of it, and how to win the non-technical hours back.

~28%of a first build goes to understanding what the source data means. The biggest single share.
~12%goes to writing the pipeline code: the part on the plan.
~60%of a first build is non-technical work: meaning, definitions, access, reconciling, meetings.
Read this first: the numbers are a rounded field estimate from many data platform projects, not a measured study. Two public surveys are quoted below for comparison. Use the picture to start a conversation in your team, then measure your own.

In one picture

Code is the tip of the iceberg

A project plan lists the things you can see: ingest, transform, test, deploy. Underneath sits the work nobody plans for, and it is bigger.

what people picture:writing the pipeline codewhat the source data meansreconciling the numbersagreeing definitionsaccess, approvals, meetingsfixing bad datareruns and backfillsthe water linewhat the plan never shows

What gets planned sits above the water. What takes the time sits below it.

The first build

Where the hours go when you onboard a source

Take one new source, say an order system, from first meeting to first trusted gold table: usually three to six months. Here is how the hours split.

Share of hours, first build of one source (about three to six months)Non-technical work: 60%Understanding what the source data meansUnderstanding what the source data means: 28% of hours in the first build28%Reconciling numbers with old reportsReconciling numbers with old reports: 10% of hours in the first build10%Agreeing definitions with the businessAgreeing definitions with the business: 8% of hours in the first build8%Getting access and approvalsGetting access and approvals: 7% of hours in the first build7%Meetings, status and handoversMeetings, status and handovers: 7% of hours in the first build7%Technical work: 40%Writing the pipeline codeWriting the pipeline code: 12% of hours in the first build12%Fixing data quality problemsFixing data quality problems: 8% of hours in the first build8%Running it: failures, reruns, backfillsRunning it: failures, reruns, backfills: 7% of hours in the first build7%Platform, CI/CD, speed and costPlatform, CI/CD, speed and cost: 7% of hours in the first build7%TestingTesting: 6% of hours in the first build6%

Understanding what the source data means takes more than twice the hours of writing the code.

Why is meaning so expensive? Because the answers are rarely written anywhere. Which column is the real key? Does a cancelled order disappear or get a status? Is updated_at set by every change or only some? Which of the three date columns is the business date? Each answer needs a person, and each wrong guess is found late, when the numbers are already in front of a user.

After go-live

A year of running looks different

First build vs a year of running the same sourcefirst builda year of runningUnderstanding what the source data meansUnderstanding what the source data means: 28% in the first build28%Understanding what the source data means: 10% in a year of running10%Reconciling numbers with old reportsReconciling numbers with old reports: 10% in the first build10%Reconciling numbers with old reports: 12% in a year of running12%Agreeing definitions with the businessAgreeing definitions with the business: 8% in the first build8%Agreeing definitions with the business: 5% in a year of running5%Getting access and approvalsGetting access and approvals: 7% in the first build7%Getting access and approvals: 3% in a year of running3%Meetings, status and handoversMeetings, status and handovers: 7% in the first build7%Meetings, status and handovers: 8% in a year of running8%Writing the pipeline codeWriting the pipeline code: 12% in the first build12%Writing the pipeline code: 8% in a year of running8%Fixing data quality problemsFixing data quality problems: 8% in the first build8%Fixing data quality problems: 20% in a year of running20%Running it: failures, reruns, backfillsRunning it: failures, reruns, backfills: 7% in the first build7%Running it: failures, reruns, backfills: 20% in a year of running20%Platform, CI/CD, speed and costPlatform, CI/CD, speed and cost: 7% in the first build7%Platform, CI/CD, speed and cost: 9% in a year of running9%TestingTesting: 6% in the first build6%Testing: 5% in a year of running5%above the dotted line: non-technical work · below: technical work

Once the pipeline runs, fixing bad data and reruns take the most hours. Hover a bar for its value.

Meaning work drops once the source is learnt, but quality fixes and reruns grow to two fifths of the time. Look at why they happen: a source team re-codes a status, adds a column, starts sending deletes differently, back-dates a correction. Those are changes in meaning that nobody announced. The technical-looking work is mostly meaning work arriving late.

Show the numbers as a table
ActivityKindFirst buildA year of runningWhat it is
Understanding what the source data meansNon-technical28%10%What one row is, which columns are the key, what each status code means, how deletes show up, which date is which, how tables join.
Reconciling numbers with old reportsNon-technical10%12%"Finance says 1,204 orders, your table says 1,191." Finding and explaining every difference.
Agreeing definitions with the businessNon-technical8%5%What counts as an active customer, a sale, a return. Often two departments disagree.
Getting access and approvalsNon-technical7%3%Credentials, firewall rules, security reviews, data-sharing sign-off, waiting.
Meetings, status and handoversNon-technical7%8%Estimates, stand-ups, status reports, explaining delays, onboarding the next person.
Writing the pipeline codeTechnical12%8%Ingestion, transforms, SCD2, gold tables, the part people picture when they hear 'data engineer'.
Fixing data quality problemsTechnical8%20%Nulls, duplicates, late files, a source that changed a column without telling anyone.
Running it: failures, reruns, backfillsTechnical7%20%A job failed at 3 a.m.; last month needs reloading; a backfill after a fix.
Platform, CI/CD, speed and costTechnical7%9%Clusters, deployments, permissions as code, tuning a slow join, the cloud bill.
TestingTechnical6%5%Unit tests on transforms, contract checks, end-to-end runs.

Why meaning is the root

One misread code, two weeks of work

Most of the expensive hours on this page trace back to one small misunderstanding about the source. Here is a real-shaped example.

misread meaningstatus 'C' taken as Closed;it means Cancelledwrong rule in codecancelled orderscounted as salesnumbers don't matchfinance reportis 3% lowerreconcile + reworktwo weeks ofmeetings and fixestrust lostusers go backto spreadsheetsone question asked early would have cost ten minutes

Reconciling, reworking and firefighting are often the bill for one question not asked early.

So the cheapest hour on any project is the one spent asking the source team the right questions before the first line of code. That is what the upstream questions and the source questionnaire are for.

Others' numbers

What public surveys say

SurveyWhat it foundHow to read it
Monte Carlo + Wakefield Research, 2022 (300 data professionals)About 40% of time spent checking or fixing data quality; about 4 hours to notice an incident and 9 to fix it.Vendor of data observability; self-reported. Close to our 'year of running' picture, where quality and reruns are the biggest shares.
dbt Labs, State of Analytics Engineering 2024 (456 people)55% name maintaining or organising data sets as a top task; 26% maintaining platforms or infrastructure.Vendor survey; 'top task', not a share of hours. Says the same thing: keeping existing data right beats building new.

Both are run by vendors who sell a fix, and both count different things from ours. They agree on the shape: keeping existing data correct and understood takes more time than building new pipelines.

The non-technical work

The non-technical work, and how to cut it

A data engineer cannot avoid non-technical work entirely; understanding the business data is part of the job. What you can avoid is doing it late, doing it again, and doing it alone. Three moves cover almost everything:

  1. Earlier. Ask before coding, request access on day one, agree the reconciliation target before building.
  2. Once. Write every answer where the next person will find it: in the metadata store next to the column, and in the semantic layer for definitions.
  3. By the right person. Chasing people, chairing meetings and settling definitions belong to an analyst, a technical anchor or the business owner. The engineer builds from written answers.
source owneranswers what the data means;announces changesbusiness ownersigns off each definitionand the reconciliationanalyst / anchorruns the questionnaire,chases answers, writes them downdata engineerbuilds, tests and runsfrom written answers

Hand each kind of work to the person whose job it is. The engineer is not the default for all of it.

Non-technical choreWhy it eats the engineer's timeHow to cut itWho should own itVerdict
Learning what the source data meansAnswers live in people's heads. The engineer finds out by trial and error, one surprise at a time, often after the code is written.Run the source questionnaire before any code. Ask the source team for 10 real records with the answers they expect. Profile the data automatically first (distinct values, nulls, key checks) so meetings start from facts. Write every answer into the metadata store, next to the column.Analyst or technical anchor asks; source owner answers; engineer reviewsShrink and do once. Never skip: it is the job
Reconciling numbers with old reportsNobody agreed what 'matching' means, so every difference becomes an investigation.Before building, agree the report to match, the tolerance (for example 0.5%) and the known differences. Build automatic reconciliation checks (counts and sums against source control totals) that run every load.Business owner signs off; engineer automates the checkPursue
Agreeing definitionsTwo departments define 'active customer' differently and the engineer is asked to pick.One named owner per metric. Definitions written once in the semantic layer (metric views) with a worked example. The engineer implements; never arbitrates.Business owner (data product owner)Pursue
Getting access and approvalsRequests start when coding starts, then the engineer waits weeks.Raise every access request on day one, in parallel, from a standard access pack. Make access-granted a gate before the build starts.Project manager or anchorPursue
Meetings and status reportsStatus is pulled from engineers by asking, in meetings.Status lives on one written plan page the engineer updates in two minutes. One weekly checkpoint. Stakeholder chasing belongs to the anchor, not the builder.Project managerPursue
Firefighting surprise source changesA source adds, renames or re-codes a column without notice; pipelines break or, worse, keep running with wrong numbers.A data contract with the source team (schema, meaning, change notice period). Schema-drift and value-drift alerts on bronze. Quarantine instead of silently loading.Source owner keeps the contract; engineer builds the alarmsPursue
Hiring more engineers to absorb itMore engineers each relearn the same meaning and attend the same meetings.Fix where the answers live instead.NobodyDrop
Letting an AI assistant guess the meaningIt drafts quickly but cannot ask the person who knows. A confident wrong guess is worse than a question.Use AI to draft questions, profile data and summarise docs; a person confirms every answer before it is written down.Analyst, with AI as helperPursue as a helper only

For the people side in depth, see How we work and the technical anchor role, who owns discovery so engineers don't have to.

Your turn

Five questions for your own team

  1. For your last source, how many weeks passed before you could say what one row means? Who answered, and where is the answer written now?
  2. Which report do your numbers have to match, within what tolerance, and who signed that off?
  3. Is there one named owner for each important definition, or does the engineer decide?
  4. How many hours last month went to reruns and fixes caused by a source change nobody announced?
  5. If your best engineer left tomorrow, which source meanings would leave with them?

FAQ

Questions people ask

What do data engineers spend most of their time on?

In our experience, understanding what the source data means: what a row is, the keys, the status codes, how deletes and dates work, how tables join. In a first build that is roughly a quarter to a third of the hours, more than writing code. Once a pipeline runs, fixing data quality problems and reruns take over, and many of those trace back to source meaning that changed or was misread.

Is the '80% of time is cleaning data' figure true?

It is folklore from data science surveys and rarely sourced well. Public surveys of data teams land nearer 40% on data quality work. The point stands: most hours go to making data right and understood, not to writing new pipelines.

How do I reduce the non-technical work as a data engineer?

Move it earlier and to the right person. Ask the source questions before coding, get access requested on day one, have a business owner sign off definitions and the reconciliation target, and write every answer down where the next person will find it. The work does not vanish, but it is done once instead of rediscovered.

Can AI understand source semantics for me?

It can profile data, read documentation and draft good questions quickly. It cannot know that status 'C' means Cancelled in this one system unless someone wrote it down. Use it to prepare; let a person who knows confirm.

Are these percentages measured?

No. They are a rounded field estimate from many data platform projects, shown to start a conversation. Measure your own: tag a month of your team's hours with these ten activities.

Sources

Checked against

Related

Where it connects

Upstreams: know your sources · Source system questionnaire · The metadata store · Metric views · Technical anchor · How we work · All data engineering