Home › Learn › Data systems, end to end

Data engineering · for data leaders first, then builders

Data systems, end to end

A data system is bigger than a data platform. It starts in systems we don't own, runs through a platform we do, and ends wherever people and programs use the numbers. This page shows the whole thing at once. Every block opens a page with the questions to chase, the decisions to take and what goes wrong in production.

In plain words

Think of a city's water

A city does not own the rivers and wells its water comes from. Each one behaves differently. One floods in the monsoon, one runs low in May, one carries silt. The treatment plant can only do its job if the engineers know these habits before they lay a pipe. At the other end, a hospital, a home and a factory all want water, but not the same water at the same pressure.

Rivers and wellseach one behavesdifferentlyTreatment plantcleaned in stages,tested at eachTapshomes, hospitals,factoriesbefore laying a pipe you ask how much the well gives, when it runs dry, and whether it is clean
A city's water supply. Sources you do not control, a plant you run, and users who each need something different.

Data works the same way. The upstreams are the source systems: apps, SaaS tools, files, APIs and events. The data platform is the plant, cleaning in stages. The downstreams are the taps: dashboards, AI, agents, ML and other apps. Most production problems that look like platform bugs began upstream, in a question nobody asked.

The whole picture

Upstreams, the data platform, downstreams

Upstreamsthe systems that create the dataApps and databasesorders, billing, HRSaaS appsCRM, ERP, payrollFilesSFTP, partner dropsAPIspartners, public dataEvents and logsclicks, IoT, queuesData platformwhat we run and are answerable forIngestionpush or pull, landingBronzeraw, as it arrivedSilverclean, one row per thingGoldshaped for a useSemanticone meaning per numberDownstreamswhere people and programs use itBI dashboardsreports, KPIsAI and agentschat with data, RAGMLfeatures, modelsAPIs and sharingapps, partnersBack to appsreverse ETLhops we own: 1 sources to bronze · 2 bronze to silver · 3 silver to gold · 4 gold to semantic · 5 semantic to experiences
The whole data system on one page. We do not own the left block, but almost every production problem starts there.

For leaders

The three blocks, in four lines each

Upstreams

The systems that create the data. We don't own them, but we depend on how they behave.

  • Each source hands data over differently: it can push changes to us, we can pull them, or it can share them in place.
  • How rows get inserted, updated and deleted decides whether a full or an incremental load is safe.
  • Keys and dates must be named per dataset, or numbers drift without anyone noticing.
  • Profiling real data, masked if needed, is how we find the edge cases before production does.
Ready Read the upstreams page

Data platform

What we build and run: ingestion, then bronze, silver, gold and the semantic layer.

  • Ingestion is driven by metadata: one framework, and a new table is configuration, not code.
  • Each layer makes one promise. Raw is kept, clean is checked, gold is reconciled and every metric has one definition.
  • Each layer has its own decisions, ways to fail in production, patterns and anti-patterns.
  • Quality, lineage, cost and access run on every hop, not as a box at the end.
Ready Read the platform page

Downstreams

Where the data earns its keep: dashboards, AI, agents, ML and apps.

  • Each consumer needs a different freshness, grain and level of trust.
  • Dashboards and AI must read the same metric definitions, or they will disagree in the same meeting.
  • Agents call typed tools over certified data. They never write raw SQL.
  • Every consumer is named with an owner, so a change upstream can be traced to who it hurts.
Ready Read the downstreams page

Drill down

Hop by hop

Each hop gets its own page with the same layout. These pages cover what to ask and what to decide. The build guide on the right covers how we build it.

#PageThe promiseStatusHow we build it
0UpstreamsKnow how each source behaves before writing a pipeline.Readyto come
·The data platform, wholeFive hops, 24 concerns, one picture; layout and environments ready.Readybuild guide
1Sources to bronzeGet everything in, exactly as it was, once.Comingbuild guide
2Bronze to silverOne clean, typed row per real thing, with history.Comingbuild guide
3Silver to goldTables shaped for a use, reconciled every run.Comingbuild guide
4Gold to semanticEvery metric defined once, with an owner.Comingbuild guide
5Semantic to experiencesDashboards, AI and agents give the same answer.Comingto come
6DownstreamsEvery consumer named, with what it needs and what it can tolerate.Readyto come

One layout

How every page is laid out

Every page has the same five parts, so a leader reads the top and a builder reads the rest. There is one question list per page, not separate lists of risks, patterns and anti-patterns. Each question carries its own reason, its production failure and its pattern.

PartWhat it holds
The promiseOne line on what this layer guarantees to the next one.
For leadersFour or five sentences. A data leader can stop reading here and still know what to ask for.
The question listOne table, grouped by category. Each row is a question to chase, why we ask it, what breaks in production if nobody asks, and the pattern to follow (with the anti-pattern to avoid).
Design decisionsThe few real choices for this layer, with the options and a verdict: Pursue, Drop, or Pursue only when a condition holds.
How we build itA link to the build guide for the same layer, so the two pages don't repeat each other.

Next

Where to go next

Start upstream: Upstreams: know your sources. It is the page to walk a client through before any build.

Hand it to a client: the printable source questionnaire.

How we build it: Data platform architecture, drawn right and Medallion architecture, drawn right.

Running the project: Technical anchor: discovery covers the whole discovery phase. These pages are its source-by-source detail.