Home › Learn › Data platform architecture

Data engineering · free guide · the whole picture

Data platform architecture, drawn right

One picture of a whole data platform, from the systems that make data to the dashboards and AI that use it: the layers in the middle, the control plane that runs them, the agents that may only suggest, and the jobs that keep every number honest. Click any box to go one level down, or run it in DataTwin Studio.

Components: Metadata store · Ingestion · Bronze to silver · Gold · Testing

In plain words

Think of a factory

The production line moves raw material through stations until it becomes a finished product; that is the row of layers in the middle, the medallion. The planning office decides what each station does, from written instructions, and keeps a log of every shift; that is the control plane. Advisors walk the floor and suggest improvements, but they cannot touch the machines; that is the agent plane. And some jobs happen at every station: inspection, safety, costing, record-keeping; those are the cross-cutting concerns. A good factory builds those in from day one, instead of adding an inspector after the first recall.

The whole picture

Planes on top, the line in the middle, concerns underneath

Read it top to bottom, then left to right. Data moves one way along the middle row. Every box with a small circle opens its own page.

Agent planeonboarding · rule suggester · triage & self-heal · cost advisor · docs · conversation UI. Agents open pull requests against the manifest; they never touch production.Control planePlatform manifest (YAML) → validator → metadata DB → generators → pipelines for your platform's binding · orchestration · catalog · run logService A · Acquire (ingest → silver)Service B · Model (gold → semantic)Service C · ServeOpen: SourcesSourcesdatabases · CDCfiles · APIsevent streamsSaaS appsOpen: LandingLandingfiles as receivednever changedshort retentionoptional for CDCOpen: BronzeBronzeraw, append-only+ ingest metadatasource schemareplayableOpen: SilverSilvertyped, dedupedconformed keys, SCDcontract enforcedbad rows quarantinedOpen: GoldGoldbusiness shapestar schema, martsdata productsreads only silverOpen: SemanticSemanticmetrics defined onceentities, joinsgoverned namesno tables of its ownOpen: ExperiencesExperiencesdashboardsconversation UIAPIs, sharingML features, RAGStreaming lane: the same layers, run continuously. Event stream → bronze stream → silver streaming tables → gold incremental views. Same contracts, same checks.Cross-cutting: built in, declared in the manifest, never bolted onX1 Metadata & controlX1 Metadata & controlX2 OrchestrationX2 OrchestrationX3 Contracts & schemaX3 Contracts & schemaX4 Data qualityX4 Data qualityX5 ProfilingX6 ReconciliationX6 ReconciliationX7 LineageX7 LineageX8 Logs, metrics, tracesX8 Logs, metrics, tracesX9 Freshness & SLOsX9 Freshness & SLOsX10 Cost & FinOpsX10 Cost & FinOpsX11 Security & accessX11 Security & accessX12 Privacy & PIIX12 Privacy & PIIX13 Governance & semanticsX13 Governance & semanticsX14 Audit & run logX14 Audit & run logX15 Replay & backfillX15 Replay & backfillX16 Environments & CI/CDX16 Environments & CI/CDX17 TestingX17 TestingX18 Resilience & DRX18 Resilience & DRX19 PerformanceX19 PerformanceX20 Self-healingX20 Self-healingX21 Agent assistanceX21 Agent assistanceX22 SimulationX22 Simulation
The data platform, drawn right. Click a layer, a band or a concern to go one level down.

Control plane

Written instructions in, pipelines out

People (and agents, by pull request) describe each source and table in a short YAML file, the platform manifest: where it comes from, how it is loaded, its key, how history is kept, its contract, its quality checks, who owns it, its budget. Nothing else is hand-written.

  1. Validate. A validator checks every key against a registry of allowed keys and types.
  2. Load the metadata DB. A loader writes the manifest into a metadata database: master tables for sources and datasets, key-value config tables, and one stable view that every pipeline reads. One database per environment (dev, test, prod), same schema. The design is described in Metadata DB Design for Data Engineering Platforms.
  3. Generate. Generators turn the metadata into pipelines for the chosen platform, using that platform's binding file.
  4. Run and record. The orchestrator runs them; every run is logged with a fingerprint of the exact config it used, so any replay or backfill can prove what it ran with.

Runtime state (watermarks, run logs, quality results) is kept apart from configuration, so changing a setting never rewrites history.

Zoom in: The metadata store: the tables, the reader view, the YAML loader and the fingerprint, with code.

Services

Three services, three owners

ServiceCoversIts jobTypical owner
A · Acquiresources → landing → bronze → silverGet every source in, faithfully and safely, and make one clean, trusted row per real thing.Platform / ingestion team
B · Modelgold → semanticShape data for the business: facts, dimensions, marts, data products and metric definitions.Analytics engineers with the business
C · ServeexperiencesPut it in front of people and programs: dashboards, conversation UIs, APIs, sharing, ML features, documents for AI.Product and BI teams

Splitting them means each can change at its own pace, and a failure in one doesn't silently corrupt another: Serve only ever reads from Model, and Model only from Acquire's silver.

Streaming

One medallion, two speeds

Streaming does not need a second architecture. Hot tables flow through the same layers continuously: event stream → bronze stream → silver streaming tables → gold incremental views. The contracts, quality checks, lineage and rules are the same; only the speed changes. Late events are handled with watermarks, and every step must still be safe to run twice.

Cross-cutting

22 concerns, built in from day one

These are the jobs that apply to every table at every hop. Each one is declared per table in the manifest and is on by default. They fall into five questions.

Trust: Can I trust the number?

X3Contracts & schemaWhat a table promises, and what happens when a source changes shape
X4Data qualityRules on rows and tables; bad rows quarantined
X5ProfilingKnowing what the data looks like, so change is visible
X6ReconciliationCounts and sums agree from source to gold

See: Can I see what happened and what it cost?

X7LineageWhere each table came from and what it feeds
X8Logs, metrics, tracesOne run is followable end to end
X9Freshness & SLOsIs the data as new as promised?
X10Cost & FinOpsWho spent what, against which budget
X19PerformanceFile sizes, layout, compaction, right-sized compute

Guard: Who may see and change what?

X11Security & accessWho may read or change what
X12Privacy & PIIFind, mask, keep only as long as allowed, erase on request
X13Governance & semanticsOwners, glossary, metric definitions
X14Audit & run logA record of every run and every change

Run: Does it run safely, every day?

X1Metadata & controlEverything is declared in one manifest and validated
X2OrchestrationWhat runs when, and in what order
X15Replay & backfillRe-running never doubles a number
X16Environments & CI/CDDev → test → prod, infrastructure as code
X17TestingUnit, contract, data and simulation tests
X18Resilience & DRRetries, retention, time travel, recovery

Self: Can it help fix itself, safely?

X20Self-healingDetect, fix in dev, deploy, verify, roll back
X21Agent assistanceAgents draft manifests, rules and docs
X22SimulationA fake world that breaks the platform on purpose

Agent plane

Agents that may only suggest

Agents help with two jobs: writing metadata, and handling incidents. Neither lets an agent change production. This is deterministic first: code that can decide something decides it, and an agent only comes in where judgement is really needed.

AgentReadsWritesNeeds a person when
Onboardingsource catalog, sample rowsa draft manifest entry and a draft contractalways, before merge
Rule suggesterprofiles, historyproposed quality rulesa rule would tighten a contract
Triage and self-healincidents, lineage, run logsa fix as a pull request, tested in devthe fix touches a contract or a metric definition
Cost advisorbilling and query historyfindings and a fix pull request (file sizes, partitions, clusters)the change affects a service level
Docsmanifest, lineagetable and metric documentationnever (docs are easy to undo)
Conversation UIthe semantic layer onlyanswers that cite a metricnever; it never writes SQL

The self-healing loop: detect → triage → fix in dev → validate → gate → pull request → deploy → verify it held → roll back if not.

Simulation: a pretend shop produces data on its own clock, and incidents are planted on purpose (a renamed column, a day sent twice, a source going dark). The platform must catch each one before production does.

Rules

Nine design rules

Each rule says what it forbids and can be checked by a machine.

1
The design names no product and no business. A bank and a clothing shop read the same architecture. Product names live only in the binding file for each platform.
2
Everything is declared in the manifest. No pipeline, table or check exists that the manifest didn't ask for. What runs is compared with what is declared.
3
Generators write pipelines; people and agents write the manifest. Generated code is never edited by hand; CI regenerates it and fails on a difference.
4
Every vendor sits behind a port. No 'if cloud == …' branches. Changing platform means a new adapter and a new binding file, never a redesign.
5
Cross-cutting jobs are on by default. Turning one off for a table needs a written waiver with a reason, an approver and an expiry date.
6
Agents propose, gates decide. Agents can only open pull requests against the manifest. Contracts and metric definitions are protected files.
7
Nothing passes by absence. A check that looked at 0 rows is an error, never a pass.
8
Data flows one way. Gold never reads bronze; a dashboard never reads silver. Lineage is checked against the allowed paths.
9
Every kind of incident has a planted test. If the simulated world can produce it, a detector must have been seen to catch it.

Concept vs product

One design, every platform

The picture above names no product. Each platform gets a binding: which product does each job. Moving platform means a new binding and new adapters, not a new design.

JobAWSAzure / FabricGoogle CloudDatabricksOpen source
Manifest → pipelinesCDK or Terraform + Glue/Step Functions definitionsBicep or Terraform + Fabric/Data Factory itemsTerraform + Composer DAGs, DataformAsset Bundles (bundles) + LakeflowPython generator + Airflow/Dagster
OrchestrationMWAA (Airflow), Step FunctionsData Factory / Fabric pipelinesCloud ComposerLakeflow JobsAirflow or Dagster
ComputeGlue, EMR, AthenaFabric Spark, SQLDataproc, BigQueryDatabricks serverless and clustersSpark, DuckDB
TablesIceberg on S3Delta in OneLakeBigQuery, Iceberg on BigLakeDelta in Unity CatalogDelta or Iceberg
Catalog, access, lineageGlue Catalog, Lake Formation, DataZoneMicrosoft PurviewDataplexUnity CatalogOpenMetadata, OpenLineage
StreamingKinesis, MSKEvent Hubs, EventstreamPub/Sub, DataflowStructured StreamingKafka + Spark
Semantic layerdbt semantic layerPower BI semantic modelsLookerMetric viewsCube Core
CostCost Explorer, CURCost ManagementBilling exportBilling system tablesquery plans and run times

Product names change often; check the current docs before you build.

One binding in full: the data platform on AWS + Databricks: the whole picture with a product in every box, the shared assets, and all 22 concerns with what Databricks gives and what we fill in.

FAQ

Questions people ask

What is a data platform architecture?

The overall design of how data gets from the systems that create it to the people, dashboards and AI that use it: the layers it passes through, the services that move it, the control plane that configures it, and the jobs such as quality, lineage, security and cost that apply everywhere.

What is the difference between the control plane and the data plane?

The data plane is where data actually moves: the layers from sources to experiences. The control plane decides what should move and how: it reads the manifest and metadata, generates the pipelines, schedules them and records every run.

Is medallion architecture the same as a data platform architecture?

No. The medallion (bronze, silver, gold) is the middle row: the layers data moves through. A full platform architecture also has the control plane above, the cross-cutting jobs below, the services that own each part, and a plan for streaming.

Why a metadata-driven platform?

Because then a new source is a few lines of YAML, not a new notebook. The manifest switches on quality, lineage, cost and access for each table, so nothing is forgotten, and every platform is built the same way.

Should AI agents be allowed to change pipelines?

Not directly. Here agents can only propose changes as pull requests against the manifest; tests and people decide. Contracts and metric definitions are protected, so an agent can never loosen a rule to make a problem disappear.

Lambda, Kappa or one medallion for streaming?

Most platforms do well with one medallion run at two speeds: the same layers and the same contracts, with hot tables flowing continuously. Kappa (everything as a stream) fits when the sources are streams to begin with.

Next

Where to go next

Zoom into the middle row: Medallion architecture, drawn right: each layer, one order followed through all of them, the rules and the mistakes.

The ideas behind the boxes: key data engineering concepts, one short page each.

On a real stack: Data platform on AWS + Databricks, and how a team builds it spec-first.

Learn it step by step: the Data Engineering path, on AWS, Azure, Google Cloud, Databricks or open source.

Each component, in detail: metadata store · ingestion · bronze to silver (SCD Type 1 and 2) · gold · test architecture.

Run it: DataTwin Studio: watch one batch go through the whole platform, recorded from a real run, then send the same file again and see what breaks.