For leaders · ways of working · data

Spec-driven development for data platforms

A small data centre of excellence (COE) writes the platform's specs once and builds the shared parts once. After that, every pipeline and every data product is built, tested, released and run from the same specs, and each new data product makes the specs stronger for the next.

Data Architectthe whole picturePlatform Architectwhat must existData Eng. leadpipelines, frameworkAnalytics Engineermetrics, semanticsDataOps and SREfreshness, costQuality and Testrules, test worldsData Stewardowners, PII, glossaryOne spec library for the data platform, versioned like code
The data COE: seven roles, one spec library. Each role writes the part only they know well.
22 concernsevery platform must handle, from quality and lineage to cost and privacy
66 rulesin the assurance catalog, each one testable
5 specsfrom "any company, any platform" down to "this company on Databricks"
8 usesfor one spec: build, test, release, operate, reconcile, audit, review, fail fast

The one-minute version. In most companies, what the data platform should do lives in people's heads and in old pipelines. Each team writes its own ingestion, quality rules are remembered rather than written, and two dashboards show two different numbers for the same thing.

With spec-driven development, that knowledge is written down once, in plain words, by the people who know it best. Pipelines are built from it, tests come from it, releases are checked against it, and auditors can trace every number back through it. When a number turns out wrong, we fix the spec first, so no later data product repeats the mistake.

Why change

Data platforms without specs, and with them

Today, without specsWith spec-driven development
Every team writes its own ingestion code for every new sourceOne ingestion framework; a new source is a few lines in the manifest
Quality rules live in people's heads, or in one notebook nobody else readsEvery rule is written in the spec and runs on every load
Two dashboards show two different revenue numbersEach metric is defined once in the semantic model, so every dashboard agrees
Cost is discovered at the end of the month, on the billEach job has a budget in the manifest, and every run records what it cost
Nobody can say where a number on the dashboard came fromLineage and the run log link every number back to its source
A late or broken source file is found by a business userDataTwin plants late and broken files before go-live; alerts fire before anyone sees a wrong number
Moving platform means rebuilding every pipelineMoving platform means writing a new binding; the manifest and the rules stay the same

Who writes the specs

The data COE team

Seven roles. Each writes the part of the spec that only they know well. Nobody has to read the whole library; each opens the part that is theirs. In a small team one person can hold two roles.

Data Architect

The whole picture: which sources, which business questions, which layers, and the main risks to the numbers.

WritesWrites the platform's purpose, its data domains and owners, and the manifest's first draft.

Platform Architect

What every platform must have underneath: storage, compute, catalog, identity, networking, metering.

WritesWrites the platform catalog (DPC) and the binding for each platform (DBS).

Data Engineering lead

How data moves from source to gold, and the shared framework every pipeline is built on.

WritesWrites the pipeline types, the ingestion and transformation patterns, and the reference framework.

Analytics Engineer

What the numbers mean: each metric defined once, with its joins and its grain (the level of detail one row stands for).

WritesWrites the semantic model: metrics, dimensions and their owners.

DataOps and SRE

How it runs every day: freshness targets, alerts, runbooks, reruns and cost limits.

WritesWrites the operations part of the manifest and the release gate checklist.

Data Quality and Test lead

How we prove the numbers are right: which rules, which tests, which faults to plant.

WritesWrites the assurance rules (DAC) and the test worlds (DWD) for DataTwin.

Data Steward

Who owns each dataset, which columns hold personal data, and what each business term means: for example, the one agreed definition of scrap rate.

WritesWrites ownership, privacy tags, retention and the glossary.

Five specs, four kinds

Some specs fit every company, some fit one platform

Two simple questions sort every spec. Does it name a product (tech binding), or not (tech agnostic)? Does it fit any company (domain agnostic), or only this company and its data (company specific)?

Catalogs (3 specs)DPC: what every platform must haveDAC: what must always be trueDWD: the world it must facewritten once, used by allManifest (DPM)What this company's platform does:its sources, datasets, contracts,owners, schedules and targetsone per companyBinding (DBS)Each concern mapped to a product:Databricks on AWS, Fabric, ...the only place products appearone per platformThe running platformPipelines and data products live,with their SLOs, alerts, runbooks,costs and the gate evidenceone per releaseTechagnosticno product namesTechbindingnames the productDomain agnosticany company, any industryCompany specificthis company, this data
Top left is written once and used by every company. Product names live only in the bottom row, so moving platform never touches the top row.
SpecIn one lineKind
DPC, Data Platform CatalogWhat every data platform must have: its layers, pipeline types and the 22 concerns, each behind a port (a fixed plug that any product can fill)Catalog
DAC, Data Assurance CatalogWhat must always be true of the data, such as "a rerun never doubles a number", each written as a rule with a testCatalog
DWD, Data World DescriptionThe world the platform must face: sources, volumes and the faults to plant, like late, duplicate or reshaped filesCatalog
DPM, Data Platform ManifestWhat this company's platform does: its sources, datasets, data contracts, owners, schedules and targets, one per companyManifest
DBS, Data Binding SpecWhich product does each job, for example Databricks on AWS; the only spec that names productsBinding

Keeping these apart is what makes them reusable. The catalogs are the same for a tyre maker and a retailer; only the manifest changes.

What the COE builds

Specs first, then shared assets, then many data products

The COE does not only write documents. It also builds the horizontal assets: the parts every data product needs, whatever its domain. Domain teams then build data products on top, one per business question.

Data products, built by domain teams (a tyre maker, say)OEE per plantScrap and yieldWarranty claimsDealer sell-outDemand planHorizontal assets, built once by the COEIngestionframeworkMetadatastoreQualityframeworkSemanticmodelConversationUICostcontrolShift-leftchecksDataTwinsimulatorSpecs, written by the COEDPCDACDWDDPM templateDBS bindingRelease gateDatabricks on AWSMicrosoft FabricOther cloudsone lakehouse underneath: the binding decides whichevery data product feeds back down
Like a building: the specs are the foundation, the shared assets are the middle floor, and each domain team adds its own data product on top.
Horizontal assetWhat it gives every data team
Common Ingestion FrameworkOne metadata-driven way to bring any source in: a new table is a few lines of configuration, not a new pipeline (ingestion, explained)
Metadata storeOne database that holds every source, dataset, rule and schedule, read by every pipeline (metadata store, explained)
Data quality frameworkRules declared in the manifest, run on every load, bad rows set aside (quarantined) instead of silently dropped
Semantic model templateEach metric defined once, so every dashboard and every question gets the same answer
Conversation UIAsk the data a question in plain words; the answer comes from the semantic model, not from a guessed query
CostSherlockFinds what each pipeline and query costs, and the patterns that waste money
Shift-left checksProblems in pipeline code and SQL caught before they run, not after (shift left, explained)
DataTwinA pretend company that sends late, duplicate and broken data on purpose, to test the platform before real data does (DataTwin, explained)
Reference data platform frameworkWorking code to copy: ports, adapters and the first slice on Databricks (the specs are open in the data platform catalog)

Not only for data engineers

One spec, eight uses

Most people think a spec is something engineers read before coding. Here the same manifest and rules also drive testing, release, daily operations, reconciliation and audit. That is where most of the saving comes from.

Buildpipelines generated from itTestevery rule becomes a testReleasethe gate checklistOperate (SRE)freshness SLOs and alertsReconcilesource and gold must agreeAuditevidence for every numberReviewcheck against it, not opinionFail fastgaps show on day oneThe manifest
Testers, reviewers, SRE, finance and auditors all work from the same manifest the pipelines were built from, so nobody works from a different idea of what 'correct' means.

Example: the release gate for a data product

Before a data product goes live, the gate is checked. Nobody writes this list by hand. It is generated from the specs, and every line says which spec it came from. If a line cannot be ticked, the release stops.

Then, the data products

How one data product gets built: thin slices

We never write the whole spec up front. We take one thin slice, such as "scrap rate, from the plant floor (MES and curing-press sensors) to one trusted number", and take it all the way through. Then the next slice.

7Every gap the slice finds is fixed in the spec firstbronzesilvergold!1Specmanifest + catalogs2Designports and layers3Buildon Databricks4DataTwina pretend month5Release gateall ticks, or stop6Data productpeople use it
Step 7 is the important one. Every gap a slice finds goes back into the specs and the shared assets.
  1. Spec. The domain team fills in the manifest for this slice: sources, datasets, contracts, owners, targets. The catalogs supply the rest.
  2. Design. The slice is mapped onto the layers and ports. Nothing new is invented if a shared asset already does the job.
  3. Build. Pipelines are built on the shared framework, on Databricks.
  4. DataTwin. A pretend company sends a month of data, with late, duplicate and broken files planted, and the numbers are checked against the pretend world's own totals.
  5. Release gate. Every line of the checklist is ticked, or the release stops.
  6. Data product. Dashboards, the semantic model and plain-language questions go live.
  7. Fix the spec. A missing rule, a new fault, a better default goes back to the COE.

Where we are

Proven on agents, in progress for data

We first used this method for AI agents. We rebuilt the same agent from the specs alone, fixed every gap it found, and rebuilt again. The gaps fell from 28 to 8, and none came back. For data platforms the specs and the first slice are written, and the first slice is being run end to end on Databricks now.

Agents: proven28Run 324Run 48Run 5gaps found rebuilding an agent from the specs aloneData platforms: in progress22 concerns, 10 pipeline types, 22 portsdone66 rules in the assurance catalogdoneslice 1 specified: 32 capabilitiesdoneslice 1 running on Databricksnowthe 30-day DataTwin proofnext
Left: measured on agents. Right: the data platform work as of 2 November 2026.

Same way as agents

The same method we use for AI agents

The data specs are siblings of the agent specs. The roles and the four kinds of spec are the same; only the subject changes.

Question the spec answersData engineeringAI engineering (agent harness)
What every system must haveData Platform Catalog (DPC)AI Harness Catalog (AHC)
What must always be trueData Assurance Catalog (DAC)AI Assurance Catalog (AAC)
The world it must faceData World Description (DWD)Agent world description
What this one system doesData Platform Manifest (DPM), one per companyAgent spec, one per agent
Which product does each jobData Binding Spec (DBS)Binding per cloud
Practice before it is realDataTwinAgentTwin
Working example to copyReference data platform frameworkReference agent and Agent Core Framework

Read more: spec-driven development for agents, data platform architecture, the data platform on AWS and Databricks, DataTwin, explained.

The case for leaders

What you get, and what it takes

For business leaders

One number for one question, because every metric is defined once.

New data products in weeks, because ingestion, quality and cost control are already built.

Audit-ready: every number can be traced to its source, its rules and its run.

No lock-in: moving platform is a new binding, not a rebuild.

For technology leaders

One way of building pipelines across teams, so people can move between them.

Testers, SRE and stewards join at the start, not after the first wrong number.

Platform features are used to the full, but always through a port, so they can be swapped.

Cost is visible per job and per data product, not only per month.

What it takes

Time up front. The first data product is slower, because the shared layer is built alongside it.

Discipline. "Fix the spec first" must be a rule, or the specs drift from the pipelines and stop being trusted.

Named owners. The COE roles need people with time set aside, not volunteers.

How to start

Onboarding in three steps

  1. 1 · Crawl: one slice

    Pick one data product that matters but is not the riskiest. Write its manifest and use the open catalogs as they are.

    Run the release gate on it, even by hand.

    You learn: where your teams' unwritten rules live.

  2. 2 · Walk: form the COE

    Name the roles. Write the binding for your platform. Move the pilot onto the shared ingestion, quality and cost assets, and run DataTwin on it.

    You get: a second data product built in much less time than the first.

  3. 3 · Run: every domain

    Domain teams build data products on the COE layer. DataTwin runs on every change, and the release gate is required for every release.

    You get: a platform that improves with every data product.

Questions

Questions leaders ask

Isn't this just waterfall with a new name?

No. We work in thin slices: write a little spec, build it, run it, and fix the spec where the run showed it was wrong. Then the next slice. The spec changes every week.

We already have Databricks. Why do we need specs?

Databricks gives excellent parts: storage, pipelines, a catalog, quality expectations, lineage, dashboards. It does not decide which rules your data must obey, who owns each dataset, or what "fresh enough" means for your business. The specs say that once, and the binding switches the Databricks features on to meet it. Our main job is joining the parts, so one run links its quality results, lineage, cost and audit.

Who owns the specs?

The COE owns the catalogs, templates and shared assets. Each domain team owns its part of the manifest. A change to a shared spec goes through the COE, like a change to shared code.

Can AI write the specs for us?

AI helps a lot with drafting: it can read existing pipelines and propose a manifest, or suggest quality rules from a profile of the data. But decisions such as who may see a compound recipe or a dealer's personal data, or what counts as a late file, belong to people who can be held accountable for them.

What if we change platform later?

Only the binding and its adapters change. The catalogs, the manifest and the tests stay as they are, because they never named a product.

How does this relate to data contracts and other standards?

We build on them, not instead of them. Data contracts follow the open ODCS standard (Open Data Contract Standard, version 3), and lineage follows OpenLineage. The specs add what those leave out: platform-wide rules, operations targets, cost, and a pretend world to test in.

Talk to us

Want to try this with your team?

The catalogs and the DataTwin method are open. If you would like to walk through this with your leadership team, or pilot it on one data product, write to basant@cleandataengineering.ai.

Spec-driven development for agents · Back to For leaders