For leaders · ways of working · data
Spec-driven development for data platforms
A small data centre of excellence (COE) writes the platform's specs once and builds the shared parts once. After that, every pipeline and every data product is built, tested, released and run from the same specs, and each new data product makes the specs stronger for the next.
The one-minute version. In most companies, what the data platform should do lives in people's heads and in old pipelines. Each team writes its own ingestion, quality rules are remembered rather than written, and two dashboards show two different numbers for the same thing.
With spec-driven development, that knowledge is written down once, in plain words, by the people who know it best. Pipelines are built from it, tests come from it, releases are checked against it, and auditors can trace every number back through it. When a number turns out wrong, we fix the spec first, so no later data product repeats the mistake.
Why change
Data platforms without specs, and with them
| Today, without specs | With spec-driven development |
|---|---|
| Every team writes its own ingestion code for every new source | One ingestion framework; a new source is a few lines in the manifest |
| Quality rules live in people's heads, or in one notebook nobody else reads | Every rule is written in the spec and runs on every load |
| Two dashboards show two different revenue numbers | Each metric is defined once in the semantic model, so every dashboard agrees |
| Cost is discovered at the end of the month, on the bill | Each job has a budget in the manifest, and every run records what it cost |
| Nobody can say where a number on the dashboard came from | Lineage and the run log link every number back to its source |
| A late or broken source file is found by a business user | DataTwin plants late and broken files before go-live; alerts fire before anyone sees a wrong number |
| Moving platform means rebuilding every pipeline | Moving platform means writing a new binding; the manifest and the rules stay the same |
Who writes the specs
The data COE team
Seven roles. Each writes the part of the spec that only they know well. Nobody has to read the whole library; each opens the part that is theirs. In a small team one person can hold two roles.
Data Architect
The whole picture: which sources, which business questions, which layers, and the main risks to the numbers.
WritesWrites the platform's purpose, its data domains and owners, and the manifest's first draft.
Platform Architect
What every platform must have underneath: storage, compute, catalog, identity, networking, metering.
WritesWrites the platform catalog (DPC) and the binding for each platform (DBS).
Data Engineering lead
How data moves from source to gold, and the shared framework every pipeline is built on.
WritesWrites the pipeline types, the ingestion and transformation patterns, and the reference framework.
Analytics Engineer
What the numbers mean: each metric defined once, with its joins and its grain (the level of detail one row stands for).
WritesWrites the semantic model: metrics, dimensions and their owners.
DataOps and SRE
How it runs every day: freshness targets, alerts, runbooks, reruns and cost limits.
WritesWrites the operations part of the manifest and the release gate checklist.
Data Quality and Test lead
How we prove the numbers are right: which rules, which tests, which faults to plant.
WritesWrites the assurance rules (DAC) and the test worlds (DWD) for DataTwin.
Data Steward
Who owns each dataset, which columns hold personal data, and what each business term means: for example, the one agreed definition of scrap rate.
WritesWrites ownership, privacy tags, retention and the glossary.
Five specs, four kinds
Some specs fit every company, some fit one platform
Two simple questions sort every spec. Does it name a product (tech binding), or not (tech agnostic)? Does it fit any company (domain agnostic), or only this company and its data (company specific)?
| Spec | In one line | Kind |
|---|---|---|
| DPC, Data Platform Catalog | What every data platform must have: its layers, pipeline types and the 22 concerns, each behind a port (a fixed plug that any product can fill) | Catalog |
| DAC, Data Assurance Catalog | What must always be true of the data, such as "a rerun never doubles a number", each written as a rule with a test | Catalog |
| DWD, Data World Description | The world the platform must face: sources, volumes and the faults to plant, like late, duplicate or reshaped files | Catalog |
| DPM, Data Platform Manifest | What this company's platform does: its sources, datasets, data contracts, owners, schedules and targets, one per company | Manifest |
| DBS, Data Binding Spec | Which product does each job, for example Databricks on AWS; the only spec that names products | Binding |
Keeping these apart is what makes them reusable. The catalogs are the same for a tyre maker and a retailer; only the manifest changes.
What the COE builds
Specs first, then shared assets, then many data products
The COE does not only write documents. It also builds the horizontal assets: the parts every data product needs, whatever its domain. Domain teams then build data products on top, one per business question.
| Horizontal asset | What it gives every data team |
|---|---|
| Common Ingestion Framework | One metadata-driven way to bring any source in: a new table is a few lines of configuration, not a new pipeline (ingestion, explained) |
| Metadata store | One database that holds every source, dataset, rule and schedule, read by every pipeline (metadata store, explained) |
| Data quality framework | Rules declared in the manifest, run on every load, bad rows set aside (quarantined) instead of silently dropped |
| Semantic model template | Each metric defined once, so every dashboard and every question gets the same answer |
| Conversation UI | Ask the data a question in plain words; the answer comes from the semantic model, not from a guessed query |
| CostSherlock | Finds what each pipeline and query costs, and the patterns that waste money |
| Shift-left checks | Problems in pipeline code and SQL caught before they run, not after (shift left, explained) |
| DataTwin | A pretend company that sends late, duplicate and broken data on purpose, to test the platform before real data does (DataTwin, explained) |
| Reference data platform framework | Working code to copy: ports, adapters and the first slice on Databricks (the specs are open in the data platform catalog) |
Not only for data engineers
One spec, eight uses
Most people think a spec is something engineers read before coding. Here the same manifest and rules also drive testing, release, daily operations, reconciliation and audit. That is where most of the saving comes from.
Example: the release gate for a data product
Before a data product goes live, the gate is checked. Nobody writes this list by hand. It is generated from the specs, and every line says which spec it came from. If a line cannot be ticked, the release stops.
- ✓The data contract is validated: column names, types, keys and allowed values match what the source promised.From: Manifest (DPM) + data contract
- ✓Every assurance rule for this dataset has a passing test, or a written waiver with a reason, an owner and an end date.From: Assurance catalog (DAC)
- ✓A freshness target is set, an alert fires when it is missed, and the alert has a runbook and a named person on call.From: Manifest (DPM), operations part
- ✓Counts and sums agree from the source to gold (reconciliation), within the agreed tolerance.From: Assurance catalog (DAC)
- ✓Personal data of dealers, fleet drivers and consumers is tagged and masked (India DPDP Act, EU GDPR), and compound recipes are readable only by a named group.From: Manifest (DPM) + steward's tags
- ✓Each job has a cost budget, and the run log records what every run cost.From: Manifest (DPM) + binding (DBS)
- ✓Replaying a day and backfilling a month were tried once, and no number doubled.From: Assurance catalog (DAC)
- ✓Lineage is visible: from any number on a dashboard you can walk back to its source tables.From: Platform catalog (DPC) + binding (DBS)
Then, the data products
How one data product gets built: thin slices
We never write the whole spec up front. We take one thin slice, such as "scrap rate, from the plant floor (MES and curing-press sensors) to one trusted number", and take it all the way through. Then the next slice.
- Spec. The domain team fills in the manifest for this slice: sources, datasets, contracts, owners, targets. The catalogs supply the rest.
- Design. The slice is mapped onto the layers and ports. Nothing new is invented if a shared asset already does the job.
- Build. Pipelines are built on the shared framework, on Databricks.
- DataTwin. A pretend company sends a month of data, with late, duplicate and broken files planted, and the numbers are checked against the pretend world's own totals.
- Release gate. Every line of the checklist is ticked, or the release stops.
- Data product. Dashboards, the semantic model and plain-language questions go live.
- Fix the spec. A missing rule, a new fault, a better default goes back to the COE.
Where we are
Proven on agents, in progress for data
We first used this method for AI agents. We rebuilt the same agent from the specs alone, fixed every gap it found, and rebuilt again. The gaps fell from 28 to 8, and none came back. For data platforms the specs and the first slice are written, and the first slice is being run end to end on Databricks now.
Same way as agents
The same method we use for AI agents
The data specs are siblings of the agent specs. The roles and the four kinds of spec are the same; only the subject changes.
| Question the spec answers | Data engineering | AI engineering (agent harness) |
|---|---|---|
| What every system must have | Data Platform Catalog (DPC) | AI Harness Catalog (AHC) |
| What must always be true | Data Assurance Catalog (DAC) | AI Assurance Catalog (AAC) |
| The world it must face | Data World Description (DWD) | Agent world description |
| What this one system does | Data Platform Manifest (DPM), one per company | Agent spec, one per agent |
| Which product does each job | Data Binding Spec (DBS) | Binding per cloud |
| Practice before it is real | DataTwin | AgentTwin |
| Working example to copy | Reference data platform framework | Reference agent and Agent Core Framework |
Read more: spec-driven development for agents, data platform architecture, the data platform on AWS and Databricks, DataTwin, explained.
The case for leaders
What you get, and what it takes
For business leaders
One number for one question, because every metric is defined once.
New data products in weeks, because ingestion, quality and cost control are already built.
Audit-ready: every number can be traced to its source, its rules and its run.
No lock-in: moving platform is a new binding, not a rebuild.
For technology leaders
One way of building pipelines across teams, so people can move between them.
Testers, SRE and stewards join at the start, not after the first wrong number.
Platform features are used to the full, but always through a port, so they can be swapped.
Cost is visible per job and per data product, not only per month.
What it takes
Time up front. The first data product is slower, because the shared layer is built alongside it.
Discipline. "Fix the spec first" must be a rule, or the specs drift from the pipelines and stop being trusted.
Named owners. The COE roles need people with time set aside, not volunteers.
How to start
Onboarding in three steps
1 · Crawl: one slice
Pick one data product that matters but is not the riskiest. Write its manifest and use the open catalogs as they are.
Run the release gate on it, even by hand.
You learn: where your teams' unwritten rules live.
2 · Walk: form the COE
Name the roles. Write the binding for your platform. Move the pilot onto the shared ingestion, quality and cost assets, and run DataTwin on it.
You get: a second data product built in much less time than the first.
3 · Run: every domain
Domain teams build data products on the COE layer. DataTwin runs on every change, and the release gate is required for every release.
You get: a platform that improves with every data product.
Questions
Questions leaders ask
Isn't this just waterfall with a new name?
No. We work in thin slices: write a little spec, build it, run it, and fix the spec where the run showed it was wrong. Then the next slice. The spec changes every week.
We already have Databricks. Why do we need specs?
Databricks gives excellent parts: storage, pipelines, a catalog, quality expectations, lineage, dashboards. It does not decide which rules your data must obey, who owns each dataset, or what "fresh enough" means for your business. The specs say that once, and the binding switches the Databricks features on to meet it. Our main job is joining the parts, so one run links its quality results, lineage, cost and audit.
Who owns the specs?
The COE owns the catalogs, templates and shared assets. Each domain team owns its part of the manifest. A change to a shared spec goes through the COE, like a change to shared code.
Can AI write the specs for us?
AI helps a lot with drafting: it can read existing pipelines and propose a manifest, or suggest quality rules from a profile of the data. But decisions such as who may see a compound recipe or a dealer's personal data, or what counts as a late file, belong to people who can be held accountable for them.
What if we change platform later?
Only the binding and its adapters change. The catalogs, the manifest and the tests stay as they are, because they never named a product.
How does this relate to data contracts and other standards?
We build on them, not instead of them. Data contracts follow the open ODCS standard (Open Data Contract Standard, version 3), and lineage follows OpenLineage. The specs add what those leave out: platform-wide rules, operations targets, cost, and a pretend world to test in.
Talk to us
Want to try this with your team?
The catalogs and the DataTwin method are open. If you would like to walk through this with your leadership team, or pilot it on one data product, write to basant@cleandataengineering.ai.