Home › Data engineering › Where the time goes
Data engineering · free guide · one page
Where data engineering time really goes, and how to win it back
Ask someone what a data engineer does and they say "writes pipelines". Watch one for a month and you see something else: most hours go to finding out what the source data means, chasing people, and making numbers match. This page shows where the time goes, why the meaning of source data sits at the root of it, and how to win the non-technical hours back.
In one picture
Code is the tip of the iceberg
A project plan lists the things you can see: ingest, transform, test, deploy. Underneath sits the work nobody plans for, and it is bigger.
What gets planned sits above the water. What takes the time sits below it.
The first build
Where the hours go when you onboard a source
Take one new source, say an order system, from first meeting to first trusted gold table: usually three to six months. Here is how the hours split.
Understanding what the source data means takes more than twice the hours of writing the code.
Why is meaning so expensive? Because the answers are rarely written anywhere. Which column is the real key? Does a cancelled order disappear or get a status? Is updated_at set by every change or only some? Which of the three date columns is the business date? Each answer needs a person, and each wrong guess is found late, when the numbers are already in front of a user.
After go-live
A year of running looks different
Once the pipeline runs, fixing bad data and reruns take the most hours. Hover a bar for its value.
Meaning work drops once the source is learnt, but quality fixes and reruns grow to two fifths of the time. Look at why they happen: a source team re-codes a status, adds a column, starts sending deletes differently, back-dates a correction. Those are changes in meaning that nobody announced. The technical-looking work is mostly meaning work arriving late.
Show the numbers as a table
| Activity | Kind | First build | A year of running | What it is |
|---|---|---|---|---|
| Understanding what the source data means | Non-technical | 28% | 10% | What one row is, which columns are the key, what each status code means, how deletes show up, which date is which, how tables join. |
| Reconciling numbers with old reports | Non-technical | 10% | 12% | "Finance says 1,204 orders, your table says 1,191." Finding and explaining every difference. |
| Agreeing definitions with the business | Non-technical | 8% | 5% | What counts as an active customer, a sale, a return. Often two departments disagree. |
| Getting access and approvals | Non-technical | 7% | 3% | Credentials, firewall rules, security reviews, data-sharing sign-off, waiting. |
| Meetings, status and handovers | Non-technical | 7% | 8% | Estimates, stand-ups, status reports, explaining delays, onboarding the next person. |
| Writing the pipeline code | Technical | 12% | 8% | Ingestion, transforms, SCD2, gold tables, the part people picture when they hear 'data engineer'. |
| Fixing data quality problems | Technical | 8% | 20% | Nulls, duplicates, late files, a source that changed a column without telling anyone. |
| Running it: failures, reruns, backfills | Technical | 7% | 20% | A job failed at 3 a.m.; last month needs reloading; a backfill after a fix. |
| Platform, CI/CD, speed and cost | Technical | 7% | 9% | Clusters, deployments, permissions as code, tuning a slow join, the cloud bill. |
| Testing | Technical | 6% | 5% | Unit tests on transforms, contract checks, end-to-end runs. |
Why meaning is the root
One misread code, two weeks of work
Most of the expensive hours on this page trace back to one small misunderstanding about the source. Here is a real-shaped example.
Reconciling, reworking and firefighting are often the bill for one question not asked early.
So the cheapest hour on any project is the one spent asking the source team the right questions before the first line of code. That is what the upstream questions and the source questionnaire are for.
Others' numbers
What public surveys say
| Survey | What it found | How to read it |
|---|---|---|
| Monte Carlo + Wakefield Research, 2022 (300 data professionals) | About 40% of time spent checking or fixing data quality; about 4 hours to notice an incident and 9 to fix it. | Vendor of data observability; self-reported. Close to our 'year of running' picture, where quality and reruns are the biggest shares. |
| dbt Labs, State of Analytics Engineering 2024 (456 people) | 55% name maintaining or organising data sets as a top task; 26% maintaining platforms or infrastructure. | Vendor survey; 'top task', not a share of hours. Says the same thing: keeping existing data right beats building new. |
Both are run by vendors who sell a fix, and both count different things from ours. They agree on the shape: keeping existing data correct and understood takes more time than building new pipelines.
The non-technical work
The non-technical work, and how to cut it
A data engineer cannot avoid non-technical work entirely; understanding the business data is part of the job. What you can avoid is doing it late, doing it again, and doing it alone. Three moves cover almost everything:
- Earlier. Ask before coding, request access on day one, agree the reconciliation target before building.
- Once. Write every answer where the next person will find it: in the metadata store next to the column, and in the semantic layer for definitions.
- By the right person. Chasing people, chairing meetings and settling definitions belong to an analyst, a technical anchor or the business owner. The engineer builds from written answers.
Hand each kind of work to the person whose job it is. The engineer is not the default for all of it.
| Non-technical chore | Why it eats the engineer's time | How to cut it | Who should own it | Verdict |
|---|---|---|---|---|
| Learning what the source data means | Answers live in people's heads. The engineer finds out by trial and error, one surprise at a time, often after the code is written. | Run the source questionnaire before any code. Ask the source team for 10 real records with the answers they expect. Profile the data automatically first (distinct values, nulls, key checks) so meetings start from facts. Write every answer into the metadata store, next to the column. | Analyst or technical anchor asks; source owner answers; engineer reviews | Shrink and do once. Never skip: it is the job |
| Reconciling numbers with old reports | Nobody agreed what 'matching' means, so every difference becomes an investigation. | Before building, agree the report to match, the tolerance (for example 0.5%) and the known differences. Build automatic reconciliation checks (counts and sums against source control totals) that run every load. | Business owner signs off; engineer automates the check | Pursue |
| Agreeing definitions | Two departments define 'active customer' differently and the engineer is asked to pick. | One named owner per metric. Definitions written once in the semantic layer (metric views) with a worked example. The engineer implements; never arbitrates. | Business owner (data product owner) | Pursue |
| Getting access and approvals | Requests start when coding starts, then the engineer waits weeks. | Raise every access request on day one, in parallel, from a standard access pack. Make access-granted a gate before the build starts. | Project manager or anchor | Pursue |
| Meetings and status reports | Status is pulled from engineers by asking, in meetings. | Status lives on one written plan page the engineer updates in two minutes. One weekly checkpoint. Stakeholder chasing belongs to the anchor, not the builder. | Project manager | Pursue |
| Firefighting surprise source changes | A source adds, renames or re-codes a column without notice; pipelines break or, worse, keep running with wrong numbers. | A data contract with the source team (schema, meaning, change notice period). Schema-drift and value-drift alerts on bronze. Quarantine instead of silently loading. | Source owner keeps the contract; engineer builds the alarms | Pursue |
| Hiring more engineers to absorb it | More engineers each relearn the same meaning and attend the same meetings. | Fix where the answers live instead. | Nobody | Drop |
| Letting an AI assistant guess the meaning | It drafts quickly but cannot ask the person who knows. A confident wrong guess is worse than a question. | Use AI to draft questions, profile data and summarise docs; a person confirms every answer before it is written down. | Analyst, with AI as helper | Pursue as a helper only |
For the people side in depth, see How we work and the technical anchor role, who owns discovery so engineers don't have to.
Your turn
Five questions for your own team
- For your last source, how many weeks passed before you could say what one row means? Who answered, and where is the answer written now?
- Which report do your numbers have to match, within what tolerance, and who signed that off?
- Is there one named owner for each important definition, or does the engineer decide?
- How many hours last month went to reruns and fixes caused by a source change nobody announced?
- If your best engineer left tomorrow, which source meanings would leave with them?
FAQ
Questions people ask
What do data engineers spend most of their time on?
In our experience, understanding what the source data means: what a row is, the keys, the status codes, how deletes and dates work, how tables join. In a first build that is roughly a quarter to a third of the hours, more than writing code. Once a pipeline runs, fixing data quality problems and reruns take over, and many of those trace back to source meaning that changed or was misread.
Is the '80% of time is cleaning data' figure true?
It is folklore from data science surveys and rarely sourced well. Public surveys of data teams land nearer 40% on data quality work. The point stands: most hours go to making data right and understood, not to writing new pipelines.
How do I reduce the non-technical work as a data engineer?
Move it earlier and to the right person. Ask the source questions before coding, get access requested on day one, have a business owner sign off definitions and the reconciliation target, and write every answer down where the next person will find it. The work does not vanish, but it is done once instead of rediscovered.
Can AI understand source semantics for me?
It can profile data, read documentation and draft good questions quickly. It cannot know that status 'C' means Cancelled in this one system unless someone wrote it down. Use it to prepare; let a person who knows confirm.
Are these percentages measured?
No. They are a rounded field estimate from many data platform projects, shown to start a conversation. Measure your own: tag a month of your team's hours with these ten activities.
Sources
Checked against
Related
Where it connects
Upstreams: know your sources · Source system questionnaire · The metadata store · Metric views · Technical anchor · How we work · All data engineering