Clean Data Engineering specs

Two open catalogs for data platforms: what parts must exist, and what must be true of the data.

Not started · lead itData EngineeringForward-deployed ready

0 of 11 tasks done

1 · Purpose

Every data team rebuilds the same platform from scratch and argues about the same gaps. We would write two neutral catalogs that any team can check itself against: a capability catalog that lists the parts a data platform must have, and an assurance catalog that lists what must be true of its data, each with a way to check it. Both would follow one path, from source systems through ingestion, bronze, silver and gold layers, a semantic model and consumption objects, to dashboards and conversational analytics. They would reuse existing open standards for data contracts and lineage rather than invent new ones.

For: Data engineers, data platform architects, analytics leads and auditors who need a shared checklist.

2 · Business benefits

A shared checklistTeams can see in one place what their platform is missing before it bites them.
Faster reviewsArchitecture and audit reviews start from a standard list instead of a blank page.
Platform-neutralThe same catalog applies on Databricks, Microsoft Fabric, AWS, GCP or an open stack.
Built on standardsIt points to existing data-contract and lineage standards, so teams adopt rather than rewrite.
Testable promisesEach 'must be true' item comes with a check, so quality claims can be shown, not just stated.

3 · Status and how to join

Not started · lead it. Nobody has started this yet. Lead it, or join the first group. Lead: Open — looking for a lead.

Good first tasks this week:

  • Write one capability entry for a stage you know well, such as ingestion.
  • Write one obligation for data freshness, with how you would check it.
  • Describe one real incident you have seen as a short replayable scenario.

The code is not public yet; joiners get access.

4 · Requirements

What it must do

How well it must do it

5 · Architecture, very high level

Data path modelNames the stages from sourcesto experiences and theCapability catalogLists what parts a dataplatform must have, stage byAssurance catalogLists what must be true of thedata and how each promise isStandards crosswalkLinks entries to thedata-contract and lineageScenario libraryHolds replayable incidents,such as a renamed column or aPlatform bindingsShows how each capability isusually met on each platform.Linter and rendererChecks every entry is completeand builds the readablestages and cona capability mcite, do not rcomplete entri
Components and what each is responsible for, with no tool names: the tools come in the next section.

Also: Data path model → Assurance catalog: stages and concerns · Assurance catalog → Scenario library: incidents that test each promise · Capability catalog → Platform bindings: how each part is met

6 · Tech stack, per platform

ComponentOpen stackDatabricksAzureAWSGCP
IngestionProposal: change capture from sources into object storage————
Bronze/Silver/GoldProposal: DuckDB or ClickHouse with dbt models————
Data contractsProposal: ODCS files checked in CI————
LineageProposal: OpenLineage events into Marquez————
Semantic modelProposal: Cube CoreProposal: Metric ViewsProposal: Power BI semantic models—Proposal: Looker modelling layer
ExperiencesProposal: Superset dashboards and a conversation UI over the semantic modelProposal: AI/BI dashboards and GenieProposal: Power BI reports and CopilotProposal: QuickSightProposal: Looker dashboards
Medallion—Proposal: Delta tables with Lakeflow pipelines———
Catalog and lineage—Proposal: Unity Catalog——Proposal: Dataplex
Platform——Proposal: Microsoft Fabric lakehouse and OneLake——
Catalog and governance——Proposal: Microsoft Purview——
Storage and tables———Proposal: S3 with Apache Iceberg tables—
Transform———Proposal: Glue or EMR—
Catalog———Proposal: Glue Data Catalog and DataZone—
Warehouse————Proposal: BigQuery

Bold with ✓ = built and running today. Everything else is the planned or possible binding: the components stay the same, only the tools change.

7 · What needs to be done

Agree the frame 0 of 3 done

TaskStatusSizeSkill
Write the data path model and the list of cross-cutting concernstodoMdata architecture
Write the entry schema for capabilities and obligationstodoSspec writing
Decide the boundary between the two catalogs, with five worked examplestodoSdata architecture

First core entries 0 of 3 done

TaskStatusSizeSkill
Draft about 20 capabilities across ingestion, bronze, silver and goldtodoLdata engineering
Draft about 20 obligations for quality, freshness, schema and lineagetodoLdata quality
Map each obligation to ODCS and OpenLineage fieldstodoMstandards

Tooling 0 of 2 done

TaskStatusSizeSkill
Build the linter that rejects incomplete entriestodoMPython or JavaScript
Build the renderer that turns the catalog into web pagestodoMweb

Scenarios and bindings 0 of 2 done

TaskStatusSizeSkill
Write five replayable incidents in the AgentTwin world formattodoMtesting
Write platform bindings for an open stack and one cloudtodoMplatform engineering

Prove it on a real platform 0 of 1 done

TaskStatusSizeSkill
Run the catalog against one small reference platform and publish the gaps foundtodoLdata engineering

Size: S = an evening or two, M = about a week part-time, L = a few weeks.

8 · Roles

RoleHow manyHours / weekWhat they do
Spec lead14–6 hours/weekOwns the frame, the schema and the boundary between the two catalogs.
Data engineer contributor32–4 hours/weekDrafts entries for the stages they know best.
Platform specialist22 hours/weekWrites the binding notes for one platform each.
Reviewer21–2 hours/weekChecks entries are plain, complete and not vendor advice in disguise.

9 · What you'll learn

Level-up questions

10 · Links