Clean Data Engineering specs
Two open catalogs for data platforms: what parts must exist, and what must be true of the data.
1 · Purpose
Every data team rebuilds the same platform from scratch and argues about the same gaps. We would write two neutral catalogs that any team can check itself against: a capability catalog that lists the parts a data platform must have, and an assurance catalog that lists what must be true of its data, each with a way to check it. Both would follow one path, from source systems through ingestion, bronze, silver and gold layers, a semantic model and consumption objects, to dashboards and conversational analytics. They would reuse existing open standards for data contracts and lineage rather than invent new ones.
2 · Business benefits
3 · Status and how to join
Good first tasks this week:
- Write one capability entry for a stage you know well, such as ingestion.
- Write one obligation for data freshness, with how you would check it.
- Describe one real incident you have seen as a short replayable scenario.
4 · Requirements
What it must do
- List each platform capability (what must exist) with a plain description and what breaks without it.
- List each data obligation (what must be true) with how it is checked and what evidence counts.
- Tag every entry by stage on the data path and by cross-cutting concern, such as quality, lineage, privacy or cost.
- Map each entry to the matching part of the open data-contract standard (ODCS) and the open lineage standard (OpenLineage) where one exists.
- Describe test scenarios in the same world format the AgentTwin simulator uses, so a planted incident can be replayed.
- Give per-platform notes showing how each capability is usually met on Databricks, Fabric, AWS, GCP and an open stack.
- Lint the catalog so that every entry has an id, a stage, a concern and, for obligations, a check.
How well it must do it
- Entries cite standards and products; they never copy their text.
- Identifiers are flat and permanent, so a citation never breaks when the catalog is reorganised.
- The spec is plain text in a public repository, readable without any special tool.
- Product-specific advice lives in separate binding files, so the core catalog does not go stale with vendor releases.
- The licence lets anyone reuse and adapt the spec with credit.
5 · Architecture, very high level
6 · Tech stack, per platform
| Component | Open stack | Databricks | Azure | AWS | GCP |
|---|---|---|---|---|---|
| Ingestion | Proposal: change capture from sources into object storage | — | — | — | — |
| Bronze/Silver/Gold | Proposal: DuckDB or ClickHouse with dbt models | — | — | — | — |
| Data contracts | Proposal: ODCS files checked in CI | — | — | — | — |
| Lineage | Proposal: OpenLineage events into Marquez | — | — | — | — |
| Semantic model | Proposal: Cube Core | Proposal: Metric Views | Proposal: Power BI semantic models | — | Proposal: Looker modelling layer |
| Experiences | Proposal: Superset dashboards and a conversation UI over the semantic model | Proposal: AI/BI dashboards and Genie | Proposal: Power BI reports and Copilot | Proposal: QuickSight | Proposal: Looker dashboards |
| Medallion | — | Proposal: Delta tables with Lakeflow pipelines | — | — | — |
| Catalog and lineage | — | Proposal: Unity Catalog | — | — | Proposal: Dataplex |
| Platform | — | — | Proposal: Microsoft Fabric lakehouse and OneLake | — | — |
| Catalog and governance | — | — | Proposal: Microsoft Purview | — | — |
| Storage and tables | — | — | — | Proposal: S3 with Apache Iceberg tables | — |
| Transform | — | — | — | Proposal: Glue or EMR | — |
| Catalog | — | — | — | Proposal: Glue Data Catalog and DataZone | — |
| Warehouse | — | — | — | — | Proposal: BigQuery |
7 · What needs to be done
Agree the frame 0 of 3 done
| Task | Status | Size | Skill |
|---|---|---|---|
| Write the data path model and the list of cross-cutting concerns | todo | M | data architecture |
| Write the entry schema for capabilities and obligations | todo | S | spec writing |
| Decide the boundary between the two catalogs, with five worked examples | todo | S | data architecture |
First core entries 0 of 3 done
| Task | Status | Size | Skill |
|---|---|---|---|
| Draft about 20 capabilities across ingestion, bronze, silver and gold | todo | L | data engineering |
| Draft about 20 obligations for quality, freshness, schema and lineage | todo | L | data quality |
| Map each obligation to ODCS and OpenLineage fields | todo | M | standards |
Tooling 0 of 2 done
| Task | Status | Size | Skill |
|---|---|---|---|
| Build the linter that rejects incomplete entries | todo | M | Python or JavaScript |
| Build the renderer that turns the catalog into web pages | todo | M | web |
Scenarios and bindings 0 of 2 done
| Task | Status | Size | Skill |
|---|---|---|---|
| Write five replayable incidents in the AgentTwin world format | todo | M | testing |
| Write platform bindings for an open stack and one cloud | todo | M | platform engineering |
Prove it on a real platform 0 of 1 done
| Task | Status | Size | Skill |
|---|---|---|---|
| Run the catalog against one small reference platform and publish the gaps found | todo | L | data engineering |
8 · Roles
| Role | How many | Hours / week | What they do |
|---|---|---|---|
| Spec lead | 1 | 4–6 hours/week | Owns the frame, the schema and the boundary between the two catalogs. |
| Data engineer contributor | 3 | 2–4 hours/week | Drafts entries for the stages they know best. |
| Platform specialist | 2 | 2 hours/week | Writes the binding notes for one platform each. |
| Reviewer | 2 | 1–2 hours/week | Checks entries are plain, complete and not vendor advice in disguise. |
9 · What you'll learn
- How a data platform is laid out end to end, from sources to dashboards.
- How data contracts (ODCS) and lineage events (OpenLineage) work.
- How to write a requirement that can be checked, not just agreed with.
- How the same capability looks on different clouds.
- How open standards are written, reviewed and versioned.
Level-up questions
- Where exactly is the line between a capability (must exist) and an obligation (must be true)?
- Should the semantic model and conversational analytics be in the first release or a later one?
- How much of the data-contract standard can we point to before we have to add our own fields?
- Which small reference platform should we test the catalog against first?