Home › Learn › Data systems › Data platform
Data systems · block 2 of 3 · for data leaders, then builders
The data platform, whole
Every row arrives once, is checked at every hop, and every number has one meaning.
The platform in one picture: five hops along the line (the medallion), and 24 jobs that cut across every hop. Each hop and each job has its own page with the questions to chase, what breaks in production and the decisions to take.
For leaders
- The platform is one line of five hops. Each hop makes one promise to the next, and the medallion layers (landing, bronze, silver, gold, semantic) are that line.
- Twenty-four jobs cut across every hop: quality checks, cost, security, orchestration, testing, environments and more. Each has one page; each hop page shows only what changes there.
- Platforms rarely fail in the layers. They fail in the cross-cutting jobs nobody owned: cost, freshness, how changes reach production, who may read what.
- Everything is declared as metadata, so adding a table is configuration and the cross-cutting jobs switch on by default.
- Streaming is a second speed on the same line, used only where a decision needs data in seconds or minutes.
In plain words
A treatment plant has stages, and jobs at every stage
Water passes through screens, settling tanks and filters before it is treated and labelled for use. That is the line. But some jobs are done at every stage: someone takes samples, meters record the flow, valves are run from one control room, and everything is maintained on a schedule. If samples were only tested at the tap, a problem would be found after the town drank it.
A data platform is the same. The line is the medallion: landing, bronze, silver, gold and semantic. The jobs at every stage are the cross-cutting concerns: quality, cost, security, orchestration, testing and the rest. This page shows both, and where each one is explained.
The whole picture
The line, the control plane, and the concerns underneath
Along the line
Five hops: the medallion
A hop is the move from one layer to the next. Landing and bronze are both in hop 1, because their single job is to get data in exactly as it was. Each hop page will follow the same template as the Upstreams page.
| # | Hop | Layers it builds | The promise | Page | How we build it |
|---|---|---|---|---|---|
| H1 | Sources to bronze | Landing, Bronze | Get everything in, exactly as it was, once. | Coming | build guide |
| H2 | Bronze to silver | Silver | One clean, typed row per real thing, with history. | Coming | build guide |
| H3 | Silver to gold | Gold | Tables shaped for a use, reconciled every run. | Coming | build guide |
| H4 | Gold to semantic | Semantic | Every metric defined once, with an owner. | Coming | build guide |
| H5 | Semantic to experiences | Experiences | Dashboards, AI and agents give the same answer. | Coming | to come |
Across the line
Where each concern lives
The rule: each concern has one page with its end-to-end view, its questions and its decisions. Each hop page has one row for every concern that changes at that hop, linking to the concern's page. Nothing is written twice.
| Scope | What it means | Where the detail goes | How many |
|---|---|---|---|
| Per hop | The concern works differently at each hop (a silver contract is not a gold contract). | Mostly on the hop pages; the concern page is a short summary. | 4 |
| Both | There is a whole-platform view and a slice at each hop. Orchestration: the dependency graph end to end, and what triggers this hop. Cost: the budget per data product, and this hop's cost drivers. | The end-to-end view on the concern page; each hop's slice on its hop page. | 11 |
| Platform-wide | Decided once and the same at every hop (environments, layout, audit). | The concern page only; hop pages just link to it. | 9 |
The master list
All 24 concerns, grouped by the question they answer
These are the 22 concerns from the architecture guide, with Performance renamed "Performance & scale" to cover big-data handling, plus two more: Layout (catalogs, schemas and naming) and Streaming. Where our page isn't written yet, "read now" goes to the closest existing page.
| # | Concern | Scope | The promise | Page |
|---|---|---|---|---|
| TrustCan I trust the number? | ||||
| X3 | Contracts & schema | Per hop | Each table says what it promises, and a source changing shape is caught on arrival. | Coming read now |
| X4 | Data quality | Per hop | Rules on rows and tables at every hop; bad rows are quarantined with a reason, never dropped. | Coming read now |
| X5 | Profiling | Per hop | We know what the data looks like, so a change in shape or spread is visible. | Coming read now |
| X6 | Reconciliation | Both | Counts and sums agree from the source to gold, every run. | Coming read now |
| SeeCan I see what happened and what it cost? | ||||
| X7 | Lineage | Both | For any table or column: where it came from and what it feeds. | Coming read now |
| X8 | Logs, metrics, traces | Platform-wide | One run can be followed from the source file to the dashboard tile. | Coming read now |
| X9 | Freshness & SLOs | Both | Each table has a promised age; the time budget is split across the hops. | Coming read now |
| X10 | Cost & FinOps | Both | Every cost has an owner and a budget, end to end and per hop. | Open Ready |
| X19 | Performance & scale | Both | Big data stays fast and cheap: file sizes, layout, merges, aggregates, right-sized compute. | Coming read now |
| GuardWho may see and change what? | ||||
| X11 | Security & access | Both | Who may read or change each layer, granted on purpose. | Coming read now |
| X12 | Privacy & PII | Both | Personal data is found, tagged, masked, kept only as long as allowed and erasable. | Coming read now |
| X13 | Governance & semantics | Per hop | Every table and metric has an owner and one definition. | Coming read now |
| X14 | Audit & run log | Platform-wide | A record of every run and every change, kept as evidence. | Coming read now |
| RunDoes it run safely, every day? | ||||
| X1 | Metadata & control | Platform-wide | Everything is declared in one manifest and validated; a new table is configuration. | Coming read now |
| X2 | Orchestration | Both | What runs when, what waits for what, and what happens when a step fails. | Coming read now |
| X15 | Replay & backfill | Both | Any load can be re-run, and history rebuilt, without doubling a number. | Coming read now |
| X16 | Environments & CI/CD | Platform-wide | Every change reaches production the same way, from git, tested, approved and reversible. | Open Ready |
| X17 | Testing | Both | Unit, contract, data and simulation tests, at every hop and end to end. | Coming read now |
| X18 | Resilience & DR | Platform-wide | Retries, retention, time travel and a tested way back after a disaster. | Coming read now |
| X23 | Layout: catalogs & schemas | Platform-wide | From a table's full name you can tell its environment, layer and domain, and who may read it. | Open Ready |
| X24 | Streaming | Both | A second speed on the same line, used only where a decision needs it. | Coming read now |
| SelfCan it help fix itself, safely? | ||||
| X20 | Self-healing | Platform-wide | Detect, fix in dev, deploy, verify, roll back: with a person approving. | Coming read now |
| X21 | Agent assistance | Platform-wide | Agents draft manifests, rules and docs; they never change production. | Coming read now |
| X22 | Simulation | Platform-wide | A pretend world that breaks the platform on purpose, before production does. | Coming read now |
The grid
Concerns by hop
A filled dot means the concern has a lot to say at that hop; an open dot means a little. Platform-wide concerns are the same at every hop, so they live on their own page.
| # | Concern | H1 Sources to bronze | H2 Bronze to silver | H3 Silver to gold | H4 Gold to semantic | H5 Semantic to experiences | End to end |
|---|---|---|---|---|---|---|---|
| TrustCan I trust the number? | |||||||
| X3 | Contracts & schema | ● | ● | ● | ○ | ○ | ○ |
| X4 | Data quality | ○ | ● | ● | ○ | ○ | |
| X5 | Profiling | ● | ● | ○ | ○ | ||
| X6 | Reconciliation | ● | ● | ● | ○ | ○ | ● |
| SeeCan I see what happened and what it cost? | |||||||
| X7 | Lineage | ○ | ○ | ○ | ○ | ○ | ● |
| X8 | Logs, metrics, traces | platform-wide: the same at every hop | ● | ||||
| X9 | Freshness & SLOs | ● | ○ | ○ | ○ | ● | ● |
| X10 | Cost & FinOps | ○ | ● | ● | ● | ○ | ● |
| X19 | Performance & scale | ● | ● | ● | ● | ○ | ● |
| GuardWho may see and change what? | |||||||
| X11 | Security & access | ○ | ○ | ● | ● | ● | ● |
| X12 | Privacy & PII | ● | ● | ○ | ○ | ● | ● |
| X13 | Governance & semantics | ○ | ● | ● | ○ | ○ | |
| X14 | Audit & run log | platform-wide: the same at every hop | ● | ||||
| RunDoes it run safely, every day? | |||||||
| X1 | Metadata & control | platform-wide: the same at every hop | ● | ||||
| X2 | Orchestration | ● | ○ | ● | ○ | ○ | ● |
| X15 | Replay & backfill | ● | ● | ● | ● | ||
| X16 | Environments & CI/CD | platform-wide: the same at every hop | ● | ||||
| X17 | Testing | ○ | ● | ● | ○ | ○ | ● |
| X18 | Resilience & DR | platform-wide: the same at every hop | ● | ||||
| X23 | Layout: catalogs & schemas | platform-wide: the same at every hop | ● | ||||
| X24 | Streaming | ● | ● | ○ | ○ | ○ | ● |
| SelfCan it help fix itself, safely? | |||||||
| X20 | Self-healing | platform-wide: the same at every hop | ● | ||||
| X21 | Agent assistance | platform-wide: the same at every hop | ● | ||||
| X22 | Simulation | platform-wide: the same at every hop | ● | ||||
Two speeds
Streaming: the same line, faster
Streaming is not a sixth layer or a second platform. A streaming table goes through the same layers with the same contracts, keys and quality checks. What changes is the trigger (each change, not a schedule), the state the job must keep (for de-duplication and late data), and the cost of compute that never stops.
Hops 1 and 2 stream most often: changes captured from a database flow into bronze and are merged into silver within minutes. Gold streams when a decision needs it, such as fraud or stock levels. The semantic layer and dashboards read the result either way. Streaming is a "both" concern, so its page will cover the end-to-end choices, and each hop page will show what changes at streaming speed. Until then, read Real-time analytics, drawn simply.
The question list
The headline questions for the whole platform
One or two questions per group, the ones a data leader should be able to answer. Each concern's page has the full list.
| # | Question to chase | Why we ask | What breaks in production if nobody asks | Do this (not this) |
|---|---|---|---|---|
| A · Trustcan I trust the number? | ||||
| A1 | Which checks run on every load, and what happens to a row that fails? | Trust is earned row by row, at every hop. | Bad rows are silently dropped or silently kept, and numbers drift for weeks. | Contract checks at each hop, failing rows quarantined with a reason. Not: dropping rows, or failing a whole load for one bad row. |
| A2 | How do we prove the gold total matches the source? | A pipeline that succeeded can still have lost rows. | The board sees revenue that is 3% short, and nobody can say where it went. | Reconcile counts and sums at every hop, every run, against source control totals. Not: 'the job is green'. |
| B · Seecan I see what happened and what it cost? | ||||
| B1 | Can we follow one run from a source file to a dashboard tile? | Incidents are solved by tracing, and tracing needs ids that travel with the data. | A wrong number takes three days and four teams to trace. | A run id and batch id on every row, plus lineage collected automatically. Not: reading job logs by hand. |
| B2 | How fresh is each table promised to be, and who is told when it isn't? | Freshness is a promise to a person, not a schedule. | Users find stale data before the team does, and stop trusting the dashboard. | A freshness target per table, split across the hops, with alerts to the owner. Not: 'it runs at 2 a.m.'. |
| B3 | Who pays for each table, and what does each hop cost? | Cost without an owner only grows. | The bill triples in a quarter and nobody can say which table or team caused it. | Tag compute and storage by data product; show cost per hop and per table. Not: one shared bill. |
| C · Guardwho may see and change what? | ||||
| C1 | Who may read each layer? | Layers are promises; reading the wrong one breaks them. | Reports are built on bronze and break whenever a source changes. | Grant by layer: only the platform reads bronze, analysts read gold and semantic. Not: one big 'data readers' group. |
| C2 | Where is personal data, and can we erase one person everywhere? | Privacy law applies to every copy, not just the source. | A deletion request can't be met because nobody knows which tables hold the person. | Tag personal data as it lands, mask in silver, erase using lineage. Not: a spreadsheet of PII tables. |
| D · Rundoes it run safely, every day? | ||||
| D1 | Is a new table configuration, or new code? | Five hundred hand-written pipelines are five hundred ways to be different. | Every pipeline handles deletes, retries and quality a little differently, and each fails in its own way. | One metadata-driven framework; a new table is a manifest entry. Not: copy-paste pipelines. |
| D2 | Can every load be re-run without doubling a number? | Re-runs are normal: failures, late data, fixes. | A retried load doubles a day's sales. | Merge on keys or replace whole partitions, so every hop is safe to repeat. Not: plain appends. |
| D3 | What runs when, and what waits for what? | Hops depend on each other and on sources that run late. | Gold builds on half-loaded silver, and the dashboard shows half a day. | Trigger each hop on its inputs being complete, not on a clock. Not: guessing gaps between scheduled times. |
| D4 | How does a change reach production? | Most incidents follow a change. | A fix made by hand in production is overwritten by the next deploy. | From git, through test, by a pipeline, with a way back. Not: edits in the production UI. |
| D5 | Which tables need streaming, and why? | Streaming costs more to run and to operate. | Always-on compute for data that is read once a morning. | Batch by default; stream a table only when a decision needs it in seconds or minutes. Not: streaming everything because it's modern. |
| E · Selfcan it help fix itself, safely? | ||||
| E1 | What may agents and automation do on the platform? | Automation that can change production can also break it. | An agent 'fixes' a pipeline in production and corrupts a table. | Agents propose changes by pull request; people approve; the normal pipeline deploys. Not: agents with production write access. |
Design decisions
The decisions that shape the whole platform
How are pipelines built?
| Option | Verdict |
|---|---|
| One metadata-driven framework; a table is a manifest entry | Pursue · Pursue. Every table gets the same handling of deletes, retries, quality and lineage. |
| A low-code tool, one flow per table | Pursue when · Pursue only for a small team with a few sources, and plan the move to metadata. |
| A hand-written pipeline per table | Drop · Drop beyond a handful of tables. It doesn't scale and every one differs. |
How many layers?
| Option | Verdict |
|---|---|
| Landing, bronze, silver, gold, semantic | Pursue · Pursue. Each makes one promise; together they cover replay, cleaning, shaping and meaning. |
| Skip landing | Pursue when · Pursue only when change data capture writes straight into bronze. |
| Extra medallions (platinum and so on) | Drop · Drop. If you need another layer, you usually need more tables in gold. |
Batch or streaming?
| Option | Verdict |
|---|---|
| Batch by default, streaming per table where needed, same layers | Pursue · Pursue. One line, two speeds, one set of rules. |
| Separate batch and streaming code paths for the same table | Drop · Drop. Two versions of the same logic will disagree. |
| Stream everything | Drop · Drop. Always-on cost and harder operations for no business gain. |
Where do the cross-cutting jobs live?
| Option | Verdict |
|---|---|
| Declared per table in the manifest, on by default | Pursue · Pursue. Nobody can forget quality or lineage for a new table. |
| Added per project when someone asks | Drop · Drop. They are added after the first incident, one table at a time. |
One platform or one per domain?
| Option | Verdict |
|---|---|
| One platform with shared services; domains own their schemas and tables | Pursue · Pursue. Shared plumbing, local ownership. |
| Each domain runs its own full stack | Pursue when · Pursue only in a large organisation with strong domain teams, and with shared standards. |
FAQ
Questions people ask
What are cross-cutting concerns in a data platform?
Jobs that apply to every table at every layer rather than to one step: data quality, lineage, cost, security, privacy, orchestration, testing, environments and so on. They are best declared per table and switched on by default, not added after go-live.
Should orchestration and cost be handled per layer or end to end?
Both. Each has an end-to-end view (the whole dependency graph, the budget per data product) and a slice at each hop (what triggers this hop, what this hop costs). We keep the end-to-end view on one page and show each hop's slice on its hop page.
Where does the medallion architecture fit?
The medallion is the line through the platform: landing and bronze in the first hop, then silver, gold and semantic. The cross-cutting concerns run across it.
Is streaming a separate architecture?
Usually not. It is a second speed on the same layers, with the same contracts, keys and checks. Only the trigger, state handling and running cost differ.
Next
Where to go next
Ready now: Layout: catalogs, schemas and naming, Environments and CI/CD and Cost and FinOps, with the Databricks cost calculator.
Before the platform: Upstreams: know your sources.
How we build it: Data platform architecture, drawn right and Medallion architecture, drawn right.
Back up: Data systems, end to end.