Home › Learn › Data systems › Data platform

Data systems · block 2 of 3 · for data leaders, then builders

The data platform, whole

Every row arrives once, is checked at every hop, and every number has one meaning.

The platform in one picture: five hops along the line (the medallion), and 24 jobs that cut across every hop. Each hop and each job has its own page with the questions to chase, what breaks in production and the decisions to take.

For leaders

In plain words

A treatment plant has stages, and jobs at every stage

Water passes through screens, settling tanks and filters before it is treated and labelled for use. That is the line. But some jobs are done at every stage: someone takes samples, meters record the flow, valves are run from one control room, and everything is maintained on a schedule. If samples were only tested at the tap, a problem would be found after the town drank it.

Screenscatch whatarrivedSettlingkept, nothingthrown awayFilterscleaned, onestandardTreatedready forone useLabelledone name,one meaningat every stage: sample tests · meters · valves run from one control room · maintenance · safety
A treatment plant has stages, and jobs that happen at every stage. A data platform has both too.

A data platform is the same. The line is the medallion: landing, bronze, silver, gold and semantic. The jobs at every stage are the cross-cutting concerns: quality, cost, security, orchestration, testing and the rest. This page shows both, and where each one is explained.

The whole picture

The line, the control plane, and the concerns underneath

Control plane: metadata manifest · orchestration · run loga new table is a few lines of configuration; the concerns below switch on by defaultUpstreamssources wedon't ownDownstreamsBI, AI, agents,ML, appsLandingfiles asreceivedBronzeraw, append-only, replayableSilverclean, typed,one row per thingGoldshaped for a use,reconciledSemanticmetrics definedonce112345Streaming lane: the same layers, continuously, for the tables that need itsame contracts, same keys, same checks; only the trigger and the cost differcircled numbers are the five hops; the medallion is the line itselfTrustCan I trust the number?Contracts & schemaData qualityProfilingReconciliationSeeCan I see what happened and what it cost?LineageLogs, metrics, tracesFreshness & SLOsCost & FinOpsPerformance & scaleGuardWho may see and change what?Security & accessPrivacy & PIIGovernance & semanticsAudit & run logRunDoes it run safely, every day?Metadata & controlOrchestrationReplay & backfillEnvironments & CI/CDTestingResilience & DRLayout: catalogs & schemasStreamingSelfCan it help fix itself, safely?Self-healingAgent assistanceSimulationCross-cutting: 24 jobs that apply at every hop, grouped by the question each one answers
The data platform, whole. The line is the medallion; the bands underneath are the concerns that run at every hop.

Along the line

Five hops: the medallion

A hop is the move from one layer to the next. Landing and bronze are both in hop 1, because their single job is to get data in exactly as it was. Each hop page will follow the same template as the Upstreams page.

#HopLayers it buildsThe promisePageHow we build it
H1Sources to bronzeLanding, BronzeGet everything in, exactly as it was, once.Comingbuild guide
H2Bronze to silverSilverOne clean, typed row per real thing, with history.Comingbuild guide
H3Silver to goldGoldTables shaped for a use, reconciled every run.Comingbuild guide
H4Gold to semanticSemanticEvery metric defined once, with an owner.Comingbuild guide
H5Semantic to experiencesExperiencesDashboards, AI and agents give the same answer.Comingto come

Across the line

Where each concern lives

The rule: each concern has one page with its end-to-end view, its questions and its decisions. Each hop page has one row for every concern that changes at that hop, linking to the concern's page. Nothing is written twice.

ScopeWhat it meansWhere the detail goesHow many
Per hopThe concern works differently at each hop (a silver contract is not a gold contract).Mostly on the hop pages; the concern page is a short summary.4
BothThere is a whole-platform view and a slice at each hop. Orchestration: the dependency graph end to end, and what triggers this hop. Cost: the budget per data product, and this hop's cost drivers.The end-to-end view on the concern page; each hop's slice on its hop page.11
Platform-wideDecided once and the same at every hop (environments, layout, audit).The concern page only; hop pages just link to it.9

The master list

All 24 concerns, grouped by the question they answer

These are the 22 concerns from the architecture guide, with Performance renamed "Performance & scale" to cover big-data handling, plus two more: Layout (catalogs, schemas and naming) and Streaming. Where our page isn't written yet, "read now" goes to the closest existing page.

#ConcernScopeThe promisePage
TrustCan I trust the number?
X3Contracts & schemaPer hopEach table says what it promises, and a source changing shape is caught on arrival.Coming read now
X4Data qualityPer hopRules on rows and tables at every hop; bad rows are quarantined with a reason, never dropped.Coming read now
X5ProfilingPer hopWe know what the data looks like, so a change in shape or spread is visible.Coming read now
X6ReconciliationBothCounts and sums agree from the source to gold, every run.Coming read now
SeeCan I see what happened and what it cost?
X7LineageBothFor any table or column: where it came from and what it feeds.Coming read now
X8Logs, metrics, tracesPlatform-wideOne run can be followed from the source file to the dashboard tile.Coming read now
X9Freshness & SLOsBothEach table has a promised age; the time budget is split across the hops.Coming read now
X10Cost & FinOpsBothEvery cost has an owner and a budget, end to end and per hop.Open Ready
X19Performance & scaleBothBig data stays fast and cheap: file sizes, layout, merges, aggregates, right-sized compute.Coming read now
GuardWho may see and change what?
X11Security & accessBothWho may read or change each layer, granted on purpose.Coming read now
X12Privacy & PIIBothPersonal data is found, tagged, masked, kept only as long as allowed and erasable.Coming read now
X13Governance & semanticsPer hopEvery table and metric has an owner and one definition.Coming read now
X14Audit & run logPlatform-wideA record of every run and every change, kept as evidence.Coming read now
RunDoes it run safely, every day?
X1Metadata & controlPlatform-wideEverything is declared in one manifest and validated; a new table is configuration.Coming read now
X2OrchestrationBothWhat runs when, what waits for what, and what happens when a step fails.Coming read now
X15Replay & backfillBothAny load can be re-run, and history rebuilt, without doubling a number.Coming read now
X16Environments & CI/CDPlatform-wideEvery change reaches production the same way, from git, tested, approved and reversible.Open Ready
X17TestingBothUnit, contract, data and simulation tests, at every hop and end to end.Coming read now
X18Resilience & DRPlatform-wideRetries, retention, time travel and a tested way back after a disaster.Coming read now
X23Layout: catalogs & schemasPlatform-wideFrom a table's full name you can tell its environment, layer and domain, and who may read it.Open Ready
X24StreamingBothA second speed on the same line, used only where a decision needs it.Coming read now
SelfCan it help fix itself, safely?
X20Self-healingPlatform-wideDetect, fix in dev, deploy, verify, roll back: with a person approving.Coming read now
X21Agent assistancePlatform-wideAgents draft manifests, rules and docs; they never change production.Coming read now
X22SimulationPlatform-wideA pretend world that breaks the platform on purpose, before production does.Coming read now

The grid

Concerns by hop

A filled dot means the concern has a lot to say at that hop; an open dot means a little. Platform-wide concerns are the same at every hop, so they live on their own page.

#ConcernH1
Sources to bronze
H2
Bronze to silver
H3
Silver to gold
H4
Gold to semantic
H5
Semantic to experiences
End to end
TrustCan I trust the number?
X3Contracts & schema●●●○○○
X4Data quality○●●○○
X5Profiling●●○○
X6Reconciliation●●●○○●
SeeCan I see what happened and what it cost?
X7Lineage○○○○○●
X8Logs, metrics, tracesplatform-wide: the same at every hop●
X9Freshness & SLOs●○○○●●
X10Cost & FinOps○●●●○●
X19Performance & scale●●●●○●
GuardWho may see and change what?
X11Security & access○○●●●●
X12Privacy & PII●●○○●●
X13Governance & semantics○●●○○
X14Audit & run logplatform-wide: the same at every hop●
RunDoes it run safely, every day?
X1Metadata & controlplatform-wide: the same at every hop●
X2Orchestration●○●○○●
X15Replay & backfill●●●●
X16Environments & CI/CDplatform-wide: the same at every hop●
X17Testing○●●○○●
X18Resilience & DRplatform-wide: the same at every hop●
X23Layout: catalogs & schemasplatform-wide: the same at every hop●
X24Streaming●●○○○●
SelfCan it help fix itself, safely?
X20Self-healingplatform-wide: the same at every hop●
X21Agent assistanceplatform-wide: the same at every hop●
X22Simulationplatform-wide: the same at every hop●

Two speeds

Streaming: the same line, faster

Streaming is not a sixth layer or a second platform. A streaming table goes through the same layers with the same contracts, keys and quality checks. What changes is the trigger (each change, not a schedule), the state the job must keep (for de-duplication and late data), and the cost of compute that never stops.

Hops 1 and 2 stream most often: changes captured from a database flow into bronze and are merged into silver within minutes. Gold streams when a decision needs it, such as fraud or stock levels. The semantic layer and dashboards read the result either way. Streaming is a "both" concern, so its page will cover the end-to-end choices, and each hop page will show what changes at streaming speed. Until then, read Real-time analytics, drawn simply.

The question list

The headline questions for the whole platform

One or two questions per group, the ones a data leader should be able to answer. Each concern's page has the full list.

#Question to chaseWhy we askWhat breaks in production if nobody asksDo this (not this)
A · Trustcan I trust the number?
A1Which checks run on every load, and what happens to a row that fails?Trust is earned row by row, at every hop.Bad rows are silently dropped or silently kept, and numbers drift for weeks.Contract checks at each hop, failing rows quarantined with a reason. Not: dropping rows, or failing a whole load for one bad row.
A2How do we prove the gold total matches the source?A pipeline that succeeded can still have lost rows.The board sees revenue that is 3% short, and nobody can say where it went.Reconcile counts and sums at every hop, every run, against source control totals. Not: 'the job is green'.
B · Seecan I see what happened and what it cost?
B1Can we follow one run from a source file to a dashboard tile?Incidents are solved by tracing, and tracing needs ids that travel with the data.A wrong number takes three days and four teams to trace.A run id and batch id on every row, plus lineage collected automatically. Not: reading job logs by hand.
B2How fresh is each table promised to be, and who is told when it isn't?Freshness is a promise to a person, not a schedule.Users find stale data before the team does, and stop trusting the dashboard.A freshness target per table, split across the hops, with alerts to the owner. Not: 'it runs at 2 a.m.'.
B3Who pays for each table, and what does each hop cost?Cost without an owner only grows.The bill triples in a quarter and nobody can say which table or team caused it.Tag compute and storage by data product; show cost per hop and per table. Not: one shared bill.
C · Guardwho may see and change what?
C1Who may read each layer?Layers are promises; reading the wrong one breaks them.Reports are built on bronze and break whenever a source changes.Grant by layer: only the platform reads bronze, analysts read gold and semantic. Not: one big 'data readers' group.
C2Where is personal data, and can we erase one person everywhere?Privacy law applies to every copy, not just the source.A deletion request can't be met because nobody knows which tables hold the person.Tag personal data as it lands, mask in silver, erase using lineage. Not: a spreadsheet of PII tables.
D · Rundoes it run safely, every day?
D1Is a new table configuration, or new code?Five hundred hand-written pipelines are five hundred ways to be different.Every pipeline handles deletes, retries and quality a little differently, and each fails in its own way.One metadata-driven framework; a new table is a manifest entry. Not: copy-paste pipelines.
D2Can every load be re-run without doubling a number?Re-runs are normal: failures, late data, fixes.A retried load doubles a day's sales.Merge on keys or replace whole partitions, so every hop is safe to repeat. Not: plain appends.
D3What runs when, and what waits for what?Hops depend on each other and on sources that run late.Gold builds on half-loaded silver, and the dashboard shows half a day.Trigger each hop on its inputs being complete, not on a clock. Not: guessing gaps between scheduled times.
D4How does a change reach production?Most incidents follow a change.A fix made by hand in production is overwritten by the next deploy.From git, through test, by a pipeline, with a way back. Not: edits in the production UI.
D5Which tables need streaming, and why?Streaming costs more to run and to operate.Always-on compute for data that is read once a morning.Batch by default; stream a table only when a decision needs it in seconds or minutes. Not: streaming everything because it's modern.
E · Selfcan it help fix itself, safely?
E1What may agents and automation do on the platform?Automation that can change production can also break it.An agent 'fixes' a pipeline in production and corrupts a table.Agents propose changes by pull request; people approve; the normal pipeline deploys. Not: agents with production write access.

Design decisions

The decisions that shape the whole platform

How are pipelines built?

OptionVerdict
One metadata-driven framework; a table is a manifest entryPursue · Pursue. Every table gets the same handling of deletes, retries, quality and lineage.
A low-code tool, one flow per tablePursue when · Pursue only for a small team with a few sources, and plan the move to metadata.
A hand-written pipeline per tableDrop · Drop beyond a handful of tables. It doesn't scale and every one differs.

How many layers?

OptionVerdict
Landing, bronze, silver, gold, semanticPursue · Pursue. Each makes one promise; together they cover replay, cleaning, shaping and meaning.
Skip landingPursue when · Pursue only when change data capture writes straight into bronze.
Extra medallions (platinum and so on)Drop · Drop. If you need another layer, you usually need more tables in gold.

Batch or streaming?

OptionVerdict
Batch by default, streaming per table where needed, same layersPursue · Pursue. One line, two speeds, one set of rules.
Separate batch and streaming code paths for the same tableDrop · Drop. Two versions of the same logic will disagree.
Stream everythingDrop · Drop. Always-on cost and harder operations for no business gain.

Where do the cross-cutting jobs live?

OptionVerdict
Declared per table in the manifest, on by defaultPursue · Pursue. Nobody can forget quality or lineage for a new table.
Added per project when someone asksDrop · Drop. They are added after the first incident, one table at a time.

One platform or one per domain?

OptionVerdict
One platform with shared services; domains own their schemas and tablesPursue · Pursue. Shared plumbing, local ownership.
Each domain runs its own full stackPursue when · Pursue only in a large organisation with strong domain teams, and with shared standards.

FAQ

Questions people ask

What are cross-cutting concerns in a data platform?

Jobs that apply to every table at every layer rather than to one step: data quality, lineage, cost, security, privacy, orchestration, testing, environments and so on. They are best declared per table and switched on by default, not added after go-live.

Should orchestration and cost be handled per layer or end to end?

Both. Each has an end-to-end view (the whole dependency graph, the budget per data product) and a slice at each hop (what triggers this hop, what this hop costs). We keep the end-to-end view on one page and show each hop's slice on its hop page.

Where does the medallion architecture fit?

The medallion is the line through the platform: landing and bronze in the first hop, then silver, gold and semantic. The cross-cutting concerns run across it.

Is streaming a separate architecture?

Usually not. It is a second speed on the same layers, with the same contracts, keys and checks. Only the trigger, state handling and running cost differ.

Next

Where to go next

Ready now: Layout: catalogs, schemas and naming, Environments and CI/CD and Cost and FinOps, with the Databricks cost calculator.

Before the platform: Upstreams: know your sources.

How we build it: Data platform architecture, drawn right and Medallion architecture, drawn right.

Back up: Data systems, end to end.