Home › Learn › Data systems, end to end
Data engineering · for data leaders first, then builders
Data systems, end to end
A data system is bigger than a data platform. It starts in systems we don't own, runs through a platform we do, and ends wherever people and programs use the numbers. This page shows the whole thing at once. Every block opens a page with the questions to chase, the decisions to take and what goes wrong in production.
In plain words
Think of a city's water
A city does not own the rivers and wells its water comes from. Each one behaves differently. One floods in the monsoon, one runs low in May, one carries silt. The treatment plant can only do its job if the engineers know these habits before they lay a pipe. At the other end, a hospital, a home and a factory all want water, but not the same water at the same pressure.
Data works the same way. The upstreams are the source systems: apps, SaaS tools, files, APIs and events. The data platform is the plant, cleaning in stages. The downstreams are the taps: dashboards, AI, agents, ML and other apps. Most production problems that look like platform bugs began upstream, in a question nobody asked.
The whole picture
Upstreams, the data platform, downstreams
For leaders
The three blocks, in four lines each
Upstreams
The systems that create the data. We don't own them, but we depend on how they behave.
- Each source hands data over differently: it can push changes to us, we can pull them, or it can share them in place.
- How rows get inserted, updated and deleted decides whether a full or an incremental load is safe.
- Keys and dates must be named per dataset, or numbers drift without anyone noticing.
- Profiling real data, masked if needed, is how we find the edge cases before production does.
Data platform
What we build and run: ingestion, then bronze, silver, gold and the semantic layer.
- Ingestion is driven by metadata: one framework, and a new table is configuration, not code.
- Each layer makes one promise. Raw is kept, clean is checked, gold is reconciled and every metric has one definition.
- Each layer has its own decisions, ways to fail in production, patterns and anti-patterns.
- Quality, lineage, cost and access run on every hop, not as a box at the end.
Downstreams
Where the data earns its keep: dashboards, AI, agents, ML and apps.
- Each consumer needs a different freshness, grain and level of trust.
- Dashboards and AI must read the same metric definitions, or they will disagree in the same meeting.
- Agents call typed tools over certified data. They never write raw SQL.
- Every consumer is named with an owner, so a change upstream can be traced to who it hurts.
Drill down
Hop by hop
Each hop gets its own page with the same layout. These pages cover what to ask and what to decide. The build guide on the right covers how we build it.
| # | Page | The promise | Status | How we build it |
|---|---|---|---|---|
| 0 | Upstreams | Know how each source behaves before writing a pipeline. | Ready | to come |
| · | The data platform, whole | Five hops, 24 concerns, one picture; layout and environments ready. | Ready | build guide |
| 1 | Sources to bronze | Get everything in, exactly as it was, once. | Coming | build guide |
| 2 | Bronze to silver | One clean, typed row per real thing, with history. | Coming | build guide |
| 3 | Silver to gold | Tables shaped for a use, reconciled every run. | Coming | build guide |
| 4 | Gold to semantic | Every metric defined once, with an owner. | Coming | build guide |
| 5 | Semantic to experiences | Dashboards, AI and agents give the same answer. | Coming | to come |
| 6 | Downstreams | Every consumer named, with what it needs and what it can tolerate. | Ready | to come |
One layout
How every page is laid out
Every page has the same five parts, so a leader reads the top and a builder reads the rest. There is one question list per page, not separate lists of risks, patterns and anti-patterns. Each question carries its own reason, its production failure and its pattern.
| Part | What it holds |
|---|---|
| The promise | One line on what this layer guarantees to the next one. |
| For leaders | Four or five sentences. A data leader can stop reading here and still know what to ask for. |
| The question list | One table, grouped by category. Each row is a question to chase, why we ask it, what breaks in production if nobody asks, and the pattern to follow (with the anti-pattern to avoid). |
| Design decisions | The few real choices for this layer, with the options and a verdict: Pursue, Drop, or Pursue only when a condition holds. |
| How we build it | A link to the build guide for the same layer, so the two pages don't repeat each other. |
Next
Where to go next
Start upstream: Upstreams: know your sources. It is the page to walk a client through before any build.
Hand it to a client: the printable source questionnaire.
How we build it: Data platform architecture, drawn right and Medallion architecture, drawn right.
Running the project: Technical anchor: discovery covers the whole discovery phase. These pages are its source-by-source detail.