Home › Learn › Data platform architecture
Data engineering · free guide · the whole picture
Data platform architecture, drawn right
One picture of a whole data platform, from the systems that make data to the dashboards and AI that use it: the layers in the middle, the control plane that runs them, the agents that may only suggest, and the jobs that keep every number honest. Click any box to go one level down, or run it in DataTwin Studio.
In plain words
Think of a factory
The production line moves raw material through stations until it becomes a finished product; that is the row of layers in the middle, the medallion. The planning office decides what each station does, from written instructions, and keeps a log of every shift; that is the control plane. Advisors walk the floor and suggest improvements, but they cannot touch the machines; that is the agent plane. And some jobs happen at every station: inspection, safety, costing, record-keeping; those are the cross-cutting concerns. A good factory builds those in from day one, instead of adding an inspector after the first recall.
The whole picture
Planes on top, the line in the middle, concerns underneath
Read it top to bottom, then left to right. Data moves one way along the middle row. Every box with a small circle opens its own page.
Control plane
Written instructions in, pipelines out
People (and agents, by pull request) describe each source and table in a short YAML file, the platform manifest: where it comes from, how it is loaded, its key, how history is kept, its contract, its quality checks, who owns it, its budget. Nothing else is hand-written.
- Validate. A validator checks every key against a registry of allowed keys and types.
- Load the metadata DB. A loader writes the manifest into a metadata database: master tables for sources and datasets, key-value config tables, and one stable view that every pipeline reads. One database per environment (dev, test, prod), same schema. The design is described in Metadata DB Design for Data Engineering Platforms.
- Generate. Generators turn the metadata into pipelines for the chosen platform, using that platform's binding file.
- Run and record. The orchestrator runs them; every run is logged with a fingerprint of the exact config it used, so any replay or backfill can prove what it ran with.
Runtime state (watermarks, run logs, quality results) is kept apart from configuration, so changing a setting never rewrites history.
Zoom in: The metadata store: the tables, the reader view, the YAML loader and the fingerprint, with code.
Services
Three services, three owners
| Service | Covers | Its job | Typical owner |
|---|---|---|---|
| A · Acquire | sources → landing → bronze → silver | Get every source in, faithfully and safely, and make one clean, trusted row per real thing. | Platform / ingestion team |
| B · Model | gold → semantic | Shape data for the business: facts, dimensions, marts, data products and metric definitions. | Analytics engineers with the business |
| C · Serve | experiences | Put it in front of people and programs: dashboards, conversation UIs, APIs, sharing, ML features, documents for AI. | Product and BI teams |
Splitting them means each can change at its own pace, and a failure in one doesn't silently corrupt another: Serve only ever reads from Model, and Model only from Acquire's silver.
Streaming
One medallion, two speeds
Streaming does not need a second architecture. Hot tables flow through the same layers continuously: event stream → bronze stream → silver streaming tables → gold incremental views. The contracts, quality checks, lineage and rules are the same; only the speed changes. Late events are handled with watermarks, and every step must still be safe to run twice.
Cross-cutting
22 concerns, built in from day one
These are the jobs that apply to every table at every hop. Each one is declared per table in the manifest and is on by default. They fall into five questions.
Trust: Can I trust the number?
| X3 | Contracts & schema | What a table promises, and what happens when a source changes shape |
| X4 | Data quality | Rules on rows and tables; bad rows quarantined |
| X5 | Profiling | Knowing what the data looks like, so change is visible |
| X6 | Reconciliation | Counts and sums agree from source to gold |
See: Can I see what happened and what it cost?
| X7 | Lineage | Where each table came from and what it feeds |
| X8 | Logs, metrics, traces | One run is followable end to end |
| X9 | Freshness & SLOs | Is the data as new as promised? |
| X10 | Cost & FinOps | Who spent what, against which budget |
| X19 | Performance | File sizes, layout, compaction, right-sized compute |
Guard: Who may see and change what?
| X11 | Security & access | Who may read or change what |
| X12 | Privacy & PII | Find, mask, keep only as long as allowed, erase on request |
| X13 | Governance & semantics | Owners, glossary, metric definitions |
| X14 | Audit & run log | A record of every run and every change |
Run: Does it run safely, every day?
| X1 | Metadata & control | Everything is declared in one manifest and validated |
| X2 | Orchestration | What runs when, and in what order |
| X15 | Replay & backfill | Re-running never doubles a number |
| X16 | Environments & CI/CD | Dev → test → prod, infrastructure as code |
| X17 | Testing | Unit, contract, data and simulation tests |
| X18 | Resilience & DR | Retries, retention, time travel, recovery |
Self: Can it help fix itself, safely?
| X20 | Self-healing | Detect, fix in dev, deploy, verify, roll back |
| X21 | Agent assistance | Agents draft manifests, rules and docs |
| X22 | Simulation | A fake world that breaks the platform on purpose |
Agent plane
Agents that may only suggest
Agents help with two jobs: writing metadata, and handling incidents. Neither lets an agent change production. This is deterministic first: code that can decide something decides it, and an agent only comes in where judgement is really needed.
| Agent | Reads | Writes | Needs a person when |
|---|---|---|---|
| Onboarding | source catalog, sample rows | a draft manifest entry and a draft contract | always, before merge |
| Rule suggester | profiles, history | proposed quality rules | a rule would tighten a contract |
| Triage and self-heal | incidents, lineage, run logs | a fix as a pull request, tested in dev | the fix touches a contract or a metric definition |
| Cost advisor | billing and query history | findings and a fix pull request (file sizes, partitions, clusters) | the change affects a service level |
| Docs | manifest, lineage | table and metric documentation | never (docs are easy to undo) |
| Conversation UI | the semantic layer only | answers that cite a metric | never; it never writes SQL |
The self-healing loop: detect → triage → fix in dev → validate → gate → pull request → deploy → verify it held → roll back if not.
Simulation: a pretend shop produces data on its own clock, and incidents are planted on purpose (a renamed column, a day sent twice, a source going dark). The platform must catch each one before production does.
Rules
Nine design rules
Each rule says what it forbids and can be checked by a machine.
Concept vs product
One design, every platform
The picture above names no product. Each platform gets a binding: which product does each job. Moving platform means a new binding and new adapters, not a new design.
| Job | AWS | Azure / Fabric | Google Cloud | Databricks | Open source |
|---|---|---|---|---|---|
| Manifest → pipelines | CDK or Terraform + Glue/Step Functions definitions | Bicep or Terraform + Fabric/Data Factory items | Terraform + Composer DAGs, Dataform | Asset Bundles (bundles) + Lakeflow | Python generator + Airflow/Dagster |
| Orchestration | MWAA (Airflow), Step Functions | Data Factory / Fabric pipelines | Cloud Composer | Lakeflow Jobs | Airflow or Dagster |
| Compute | Glue, EMR, Athena | Fabric Spark, SQL | Dataproc, BigQuery | Databricks serverless and clusters | Spark, DuckDB |
| Tables | Iceberg on S3 | Delta in OneLake | BigQuery, Iceberg on BigLake | Delta in Unity Catalog | Delta or Iceberg |
| Catalog, access, lineage | Glue Catalog, Lake Formation, DataZone | Microsoft Purview | Dataplex | Unity Catalog | OpenMetadata, OpenLineage |
| Streaming | Kinesis, MSK | Event Hubs, Eventstream | Pub/Sub, Dataflow | Structured Streaming | Kafka + Spark |
| Semantic layer | dbt semantic layer | Power BI semantic models | Looker | Metric views | Cube Core |
| Cost | Cost Explorer, CUR | Cost Management | Billing export | Billing system tables | query plans and run times |
Product names change often; check the current docs before you build.
One binding in full: the data platform on AWS + Databricks: the whole picture with a product in every box, the shared assets, and all 22 concerns with what Databricks gives and what we fill in.
FAQ
Questions people ask
What is a data platform architecture?
The overall design of how data gets from the systems that create it to the people, dashboards and AI that use it: the layers it passes through, the services that move it, the control plane that configures it, and the jobs such as quality, lineage, security and cost that apply everywhere.
What is the difference between the control plane and the data plane?
The data plane is where data actually moves: the layers from sources to experiences. The control plane decides what should move and how: it reads the manifest and metadata, generates the pipelines, schedules them and records every run.
Is medallion architecture the same as a data platform architecture?
No. The medallion (bronze, silver, gold) is the middle row: the layers data moves through. A full platform architecture also has the control plane above, the cross-cutting jobs below, the services that own each part, and a plan for streaming.
Why a metadata-driven platform?
Because then a new source is a few lines of YAML, not a new notebook. The manifest switches on quality, lineage, cost and access for each table, so nothing is forgotten, and every platform is built the same way.
Should AI agents be allowed to change pipelines?
Not directly. Here agents can only propose changes as pull requests against the manifest; tests and people decide. Contracts and metric definitions are protected, so an agent can never loosen a rule to make a problem disappear.
Lambda, Kappa or one medallion for streaming?
Most platforms do well with one medallion run at two speeds: the same layers and the same contracts, with hot tables flowing continuously. Kappa (everything as a stream) fits when the sources are streams to begin with.
Next
Where to go next
Zoom into the middle row: Medallion architecture, drawn right: each layer, one order followed through all of them, the rules and the mistakes.
The ideas behind the boxes: key data engineering concepts, one short page each.
On a real stack: Data platform on AWS + Databricks, and how a team builds it spec-first.
Learn it step by step: the Data Engineering path, on AWS, Azure, Google Cloud, Databricks or open source.
Each component, in detail: metadata store · ingestion · bronze to silver (SCD Type 1 and 2) · gold · test architecture.
Run it: DataTwin Studio: watch one batch go through the whole platform, recorded from a real run, then send the same file again and see what breaks.