cleandataengineering.aiData Platform, from the big picture to the code
You are here: Start here

Data PlatformZoom level 0

A data platform, explained from the big picture down to the code

This is a guided tour of a working data platform. It starts with one picture and ends at the line of code that does the work. You can stop at any depth and still have a true picture.

Looking for the code?

Start from the code map: the whole system on one page, upstream, data platform and downstream, with every part's starting function, every numbered concern, and for every function a list of who calls it.

The questions these pages answer

The questionWhere the answer is
What happens to one day of data, from the source system to a number on a dashboard?One day, end to end
How can one piece of code load hundreds of different tables without a job written for each?Metadata store
How does data get in, whether it is pushed as files, pulled from a database, or streamed?Ingestion
How is raw data turned into data you can trust: cleaned, typed, with no duplicates?The thirteen stages
How is history kept? What are type 1 and type 2, and what happens when a change arrives late?12 History
What happens when a row is deleted at the source?Deletes
How do the tables that answer business questions get built?Silver to gold
How does a metric keep one meaning for everyone?Semantic layer
How do we know when something went wrong, and where?Operations
How do we know the numbers are right?Tests and proof

The one picture

The top row is the path the data takes. The highlighted bar is what drives every step. Click any box to zoom in.

Five words carry the whole design. Landing is where files arrive, untouched. Bronze keeps every batch exactly as received, so nothing is ever lost. Silver is the cleaned, trusted copy, one row per business key or per version. Gold is shaped for questions: dimensions and facts. The semantic layer gives each metric one definition.

Under all of it sits the metadata store, a small database. It holds the settings for every dataset and a record of everything every run did. The code is the same for every dataset. Only the settings differ. That is what metadata-driven means.

How to read these pages

Zoom levelWhat you getTime
0This page: the questions and the one picture2 minutes
1One page per layer, and one day end to end: what the layer is for, its parts, its design decisions10 minutes each
2One page per part: how it works step by step, a worked example, the settings that drive it, the real code10 to 20 minutes each
3One page per stage or per kind of history: the function line by line, and its testsAs deep as you want

Every page shows its zoom level at the top. Boxes in pictures are links: click one to zoom in. Every code box is cut from the real file when the site is built, names the file and line, and links to that line in the public repository. Every layer page ends with an honest list of what is not built yet.

The example used everywhere

The pages use one made-up company: a tyre maker with a sales system, a factory system, a laboratory, dealers, and presses that send readings. It has nine datasets, chosen so that between them they need every way of loading and every kind of history.

DatasetWhat it isHow it arrivesHistory kept in silverHow deletes are seen
Sales order lineOne line of a customer orderPushed as a change feedType 2: every versionThe feed says so
DeliveryA shipment against an orderPushed as a change feedType 1: latest stateThe feed says so
ProductionTyres made, per press and shiftPushed as a change feedAppended eventsNot expected
MaterialThe catalogue of tyresPulled as a full copyType 2: every versionA key missing from the copy
PressThe machines that cure tyresPulled as a full copyType 2: every versionA key missing from the copy
DealerWho sells the tyresPulled page by page from an APIType 2: every versionNot detected
Inspection resultLaboratory test resultsPushed as filesAppended eventsNot expected
Dealer sell-outMonthly sales reported by dealersPushed as filesSnapshot: the newest file for a month winsNot expected
Press readingSensor readings from the pressesStreamedAppended eventsNot expected

These settings are not typed here from memory. They are in the dataset manifests, for example data-platform-catalog/slices/02-tyre/manifest/datasets/erp.sales_order_line.yaml.

Where the code is

FolderWhat it holds
src/dataplatform, tests, realms, scriptsThe platform code these pages explain, its tests, and the deployable bundles. This explainer is built from them.
explainerThe generator that wrote these pages. Run make explainer to rebuild them after a change.
The specification catalog, a separate checkoutThe specifications the code follows, and the tyre example's manifests and contracts. Set DP_CATALOG to its folder.

You can also ask the platform to explain things without opening a browser. python scripts/explain.py run, metadata, cicd and code print plain-language text from a run's files, a dataset's manifest, the repository's checks, and the code itself. They use no model and no network, so the same inputs always give the same text.

What is real and what is not

The code is real and runs. The tyre company is simulated: a program plays the company day by day, sends its data through the platform, and knows the true answer for every day. That is how each layer is checked. The source connectors used in these runs are local stand-ins, not connections to real business systems. Each layer page says what is not built yet.

Look inside