Data PlatformZoom level 0
A data platform, explained from the big picture down to the code
This is a guided tour of a working data platform. It starts with one picture and ends at the line of code that does the work. You can stop at any depth and still have a true picture.
Start from the code map: the whole system on one page, upstream, data platform and downstream, with every part's starting function, every numbered concern, and for every function a list of who calls it.
The questions these pages answer
| The question | Where the answer is |
|---|---|
| What happens to one day of data, from the source system to a number on a dashboard? | One day, end to end |
| How can one piece of code load hundreds of different tables without a job written for each? | Metadata store |
| How does data get in, whether it is pushed as files, pulled from a database, or streamed? | Ingestion |
| How is raw data turned into data you can trust: cleaned, typed, with no duplicates? | The thirteen stages |
| How is history kept? What are type 1 and type 2, and what happens when a change arrives late? | 12 History |
| What happens when a row is deleted at the source? | Deletes |
| How do the tables that answer business questions get built? | Silver to gold |
| How does a metric keep one meaning for everyone? | Semantic layer |
| How do we know when something went wrong, and where? | Operations |
| How do we know the numbers are right? | Tests and proof |
The one picture
Five words carry the whole design. Landing is where files arrive, untouched. Bronze keeps every batch exactly as received, so nothing is ever lost. Silver is the cleaned, trusted copy, one row per business key or per version. Gold is shaped for questions: dimensions and facts. The semantic layer gives each metric one definition.
Under all of it sits the metadata store, a small database. It holds the settings for every dataset and a record of everything every run did. The code is the same for every dataset. Only the settings differ. That is what metadata-driven means.
How to read these pages
| Zoom level | What you get | Time |
|---|---|---|
| 0 | This page: the questions and the one picture | 2 minutes |
| 1 | One page per layer, and one day end to end: what the layer is for, its parts, its design decisions | 10 minutes each |
| 2 | One page per part: how it works step by step, a worked example, the settings that drive it, the real code | 10 to 20 minutes each |
| 3 | One page per stage or per kind of history: the function line by line, and its tests | As deep as you want |
Every page shows its zoom level at the top. Boxes in pictures are links: click one to zoom in. Every code box is cut from the real file when the site is built, names the file and line, and links to that line in the public repository. Every layer page ends with an honest list of what is not built yet.
The example used everywhere
The pages use one made-up company: a tyre maker with a sales system, a factory system, a laboratory, dealers, and presses that send readings. It has nine datasets, chosen so that between them they need every way of loading and every kind of history.
| Dataset | What it is | How it arrives | History kept in silver | How deletes are seen |
|---|---|---|---|---|
| Sales order line | One line of a customer order | Pushed as a change feed | Type 2: every version | The feed says so |
| Delivery | A shipment against an order | Pushed as a change feed | Type 1: latest state | The feed says so |
| Production | Tyres made, per press and shift | Pushed as a change feed | Appended events | Not expected |
| Material | The catalogue of tyres | Pulled as a full copy | Type 2: every version | A key missing from the copy |
| Press | The machines that cure tyres | Pulled as a full copy | Type 2: every version | A key missing from the copy |
| Dealer | Who sells the tyres | Pulled page by page from an API | Type 2: every version | Not detected |
| Inspection result | Laboratory test results | Pushed as files | Appended events | Not expected |
| Dealer sell-out | Monthly sales reported by dealers | Pushed as files | Snapshot: the newest file for a month wins | Not expected |
| Press reading | Sensor readings from the presses | Streamed | Appended events | Not expected |
These settings are not typed here from memory. They are in the dataset manifests, for example data-platform-catalog/slices/02-tyre/manifest/datasets/erp.sales_order_line.yaml.
Where the code is
| Folder | What it holds |
|---|---|
src/dataplatform, tests, realms, scripts | The platform code these pages explain, its tests, and the deployable bundles. This explainer is built from them. |
explainer | The generator that wrote these pages. Run make explainer to rebuild them after a change. |
| The specification catalog, a separate checkout | The specifications the code follows, and the tyre example's manifests and contracts. Set DP_CATALOG to its folder. |
You can also ask the platform to explain things without opening a browser. python scripts/explain.py run, metadata, cicd and code print plain-language text from a run's files, a dataset's manifest, the repository's checks, and the code itself. They use no model and no network, so the same inputs always give the same text.
The code is real and runs. The tyre company is simulated: a program plays the company day by day, sends its data through the platform, and knows the true answer for every day. That is how each layer is checked. The source connectors used in these runs are local stand-ins, not connections to real business systems. Each layer page says what is not built yet.