Home › Learn › Medallion architecture
Data engineering · free guide · the same on every platform
Medallion architecture, drawn right
Part of Data platform architecture, drawn right: this page zooms into its middle row.
The most common way to organise a data platform: raw data in, clean data in the middle, business-ready data out, and each step simple enough to check and redo. This guide shows each layer, follows one order through all of them with real rows, and lists the rules, the mistakes and the products on each platform.
In plain words
Think of a kitchen
A restaurant never cooks straight from the delivery bag. Groceries go into a store room as they arrived, labelled with when they came. On the prep counter they are washed, chopped and the bad bits thrown out. Then they are cooked into a dish for one purpose, and the menu gives each dish one name and one price. If a dish goes wrong, the cook goes back to the store room, not to the supplier.
A data platform works the same way. Bronze is the store room: everything as it arrived. Silver is the prep counter: cleaned, one of each thing. Gold is the dish: shaped for a use. The semantic layer is the menu: one meaning per number. The name "medallion" just means bronze, silver and gold, in order of polish.
The whole picture
The spine, with what sits above and below it
Read it left to right. Data moves one way. Above the layers sits the control plane, which turns a written description of each table (the manifest) into pipelines. Below them sit the cross-cutting jobs such as quality, lineage and cost, which run on every hop, not as a box at the end.
Each layer
Its job and its promise
Landing
Files exactly as they arrived, so a bad parse can be redone.
- Holds
- Files from the mover: CSV, JSON, Parquet, exports, API pages, untouched.
- Writes
- New files only. Never edited, never deleted by a pipeline.
- Read by
- The bronze loader only.
- Keep for
- Short: days or weeks. Long enough to replay a bad load.
- Skip it when
- Change data capture writes straight into bronze.
Bronze
Raw data in the source's own shape, append-only, with notes on where each row came from.
- Holds
- Every row ever received, in the source's columns and types (often all strings).
- Adds
_ingested_at,_source,_batch_id,_file: when, from where, in which batch, from which file.- Writes
- Append only. Loading the same batch again is detected by
_batch_idand skipped. - Read by
- Silver jobs only. No dashboard, no analyst, no model reads bronze.
- Promise
- You can always rebuild silver from bronze.
Silver
The first layer with an opinion: one clean row per real thing.
- Holds
- One row per business entity (an order, a customer), typed, de-duplicated, with conformed keys.
- Does
- Cast types, trim, standardise codes, fix time zones, handle nulls, keep the latest row per key, keep history (SCD Type 1 or 2).
- Enforces
- The data contract. A row that fails goes to quarantine with its reason; it is never silently dropped.
- Writes
- MERGE on the business key, so re-running changes nothing.
- Read by
- Gold jobs, data scientists, feature pipelines.
Gold
Tables shaped for a use: a sales mart, a star schema, a feature table, a data product.
- Holds
- Facts and dimensions, aggregates, marts, published data products.
- Built from
- Silver only, never bronze.
- Checked by
- Reconciliation: counts and sums agree with silver and with the source, within a tolerance.
- Read by
- The semantic layer, dashboards, sharing, ML.
Semantic
Definitions, not tables: each metric means one thing.
- Holds
- Metric definitions (net revenue = gross minus refunds, excluding test orders), entities, joins, governed names.
- Read by
- Dashboards and conversation UIs alike, so both give the same answer. The model never writes raw SQL.
- Why
- It ends the 'why do two dashboards show two revenues' meeting.
Experiences
Where people and programs use the data.
- Includes
- Dashboards, conversation UIs, APIs, data sharing, ML features, documents for AI (RAG).
- Reads
- Only from semantic or gold.
Worked example
One order, through every layer
A clothing shop sends a small file at 9 a.m. It has three rows, and two problems in it. Watch what each layer does with them.
1 · Landing: the file as it arrived
orders_2026-10-27_0900.json
{"order_id":"1001","customer":"C-77","amount":"1,250.00","currency":"inr","placed":"27/10/2026 08:58","status":"PAID "}
{"order_id":"1002","customer":null,"amount":"-40","currency":"INR","placed":"27/10/2026 08:59","status":"paid"}
{"order_id":"1001","customer":"C-77","amount":"1,250.00","currency":"inr","placed":"27/10/2026 08:58","status":"PAID "}
Order 1001 was sent twice. Order 1002 has no customer and a negative amount. Landing keeps all of it, untouched.
2 · Bronze: the same rows, plus where they came from
| order_id | customer | amount | currency | placed | status | _ingested_at | _source | _batch_id | _file |
|---|---|---|---|---|---|---|---|---|---|
| 1001 | C-77 | 1,250.00 | inr | 27/10/2026 08:58 | "PAID " | 2026-10-27T03:31:02Z | shop_db | b-0912 | orders_…0900.json |
| 1002 | null | -40 | INR | 27/10/2026 08:59 | paid | 2026-10-27T03:31:02Z | shop_db | b-0912 | orders_…0900.json |
| 1001 | C-77 | 1,250.00 | inr | 27/10/2026 08:58 | "PAID " | 2026-10-27T03:31:02Z | shop_db | b-0912 | orders_…0900.json |
Still strings (note the stray space after "PAID"), still the duplicate, still the bad row. Bronze is a faithful record, not a clean one. The green columns are the audit columns the loader adds.
3 · Silver: one clean, typed row per order
| order_id | customer_key | amount_inr | placed_at_utc | status | _batch_id |
|---|---|---|---|---|---|
| 1001 | C-77 | 1250.00 | 2026-10-27 03:28:00 | paid | b-0912 |
| quarantine: order_id | rule failed | _batch_id |
|---|---|---|
| 1002 | customer_key is required; amount must be ≥ 0 | b-0912 |
The duplicate collapsed to one row (latest per key). Text became numbers and timestamps, India time became UTC, codes were standardised. Order 1002 failed the contract and went to quarantine with its reason, where the owner can see it.
4 · Gold: shaped for the question "how much did we sell?"
| fact_sales: date_key | store_key | customer_key | order_id | amount_inr |
|---|---|---|---|---|
| 20261027 | S-04 | C-77 | 1001 | 1250.00 |
| daily_sales: date | orders | revenue_inr | reconciled |
|---|---|---|---|
| 2026-10-27 | 1 | 1250.00 | ✓ 2 distinct orders in the source = 1 in silver + 1 in quarantine |
5 · Semantic: one meaning for "revenue"
metric: net_revenue meaning: paid order amounts minus refunds, excluding test orders from: gold.fact_sales owner: finance
The dashboard and the conversation UI both ask for net_revenue, so both say ₹1,250 for 27 October, and both can say why order 1002 is missing.
Rules
Nine rules that keep it honest
Each rule can be checked by a machine, which is what keeps a medallion from slowly turning back into a swamp.
Mistakes
Common mistakes, and the fix
| Mistake | Why it hurts | Fix |
|---|---|---|
| Dashboards reading bronze | Every report cleans the data its own way, so numbers disagree | Point reports at gold or semantic; block bronze access for analysts |
| Cleaning before landing a raw copy | A bug in the cleaning has nothing to re-run from | Land and keep the raw copy first (ELT) |
| Dropping bad rows silently | Totals are quietly wrong and nobody knows why | Quarantine with a reason and alert the owner |
| INSERT instead of MERGE in silver | Every retry adds duplicates | MERGE on the business key; skip seen batches in bronze |
| One giant flat table per report in gold | The same product is spelled three ways | Facts and conformed dimensions; marts built from them |
| Too many layers ('platinum', 'diamond', 'staging2') | Nobody knows which one to read | Keep the meaning of each layer; add tables, not layers |
| Medallion per team, each with its own rules | The same customer cleaned five ways | One silver per business entity, owned and shared |
| Metric logic in dashboards | Two revenues in one meeting | Define the metric once in the semantic layer |
Concept vs product
The same layers on each platform
The layers and rules never change. Only the products that do each job change. Learn the job once; the product is a binding.
| Job | AWS | Azure / Fabric | Google Cloud | Databricks | Open source |
|---|---|---|---|---|---|
| Storage | Amazon S3 | OneLake / ADLS Gen2 | Cloud Storage | Unity Catalog volumes on cloud storage | MinIO or local disk |
| Table format | Apache Iceberg (S3 Tables) | Delta Lake | Iceberg on BigLake | Delta Lake | Delta Lake or Iceberg |
| Into landing | AWS DMS, AppFlow | Data Factory copy, mirroring | Datastream | Lakeflow Connect | dlt or Airbyte |
| Landing → bronze | Glue (Spark) or EMR | Fabric Spark / notebooks | Dataproc (Spark) | Auto Loader / Lakeflow pipelines | PySpark |
| Bronze → silver → gold | Glue, EMR, or dbt on Athena/Redshift | Fabric Spark or dbt on Fabric | Dataform or dbt on BigQuery | Lakeflow Declarative Pipelines or dbt | PySpark or dbt + DuckDB |
| Quality checks | Glue Data Quality | Purview data quality | Dataplex data quality | Pipeline expectations | Great Expectations or Soda |
| Catalog, access, lineage | Glue Data Catalog + Lake Formation | Microsoft Purview | Dataplex | Unity Catalog | OpenMetadata + OpenLineage |
| Semantic layer | dbt semantic layer | Power BI semantic models | Looker (LookML) | Metric views | Cube Core |
| Orchestration | MWAA (Airflow), Step Functions | Data Factory / Fabric pipelines | Cloud Composer | Lakeflow Jobs | Airflow or Dagster |
Product names change often; check the current docs before you build.
FAQ
Questions people ask
What is medallion architecture?
A way to organise a data platform into layers of increasing quality: bronze keeps raw data as it arrived, silver holds clean, typed, de-duplicated data with one row per real thing, and gold holds tables shaped for a business use. Each layer is built only from the one before it.
Is medallion architecture only for Databricks?
No. The name was popularised by Databricks, and Microsoft Fabric uses it too, but the idea (raw, then cleaned, then modelled) is much older and works on AWS, Google Cloud, Snowflake or open source. Only the product names change.
Is medallion a data model?
No. It says how data moves and what each layer promises. You still choose a data model inside the layers: for example dimensional (star schema) in gold, or data vault in silver.
How many layers should there be?
Three named layers (bronze, silver, gold), often with a landing area before bronze and a semantic layer after gold. If you feel you need a fourth medallion, you usually need more tables, not another layer.
Where should data quality checks go?
Everywhere, but the contract is enforced at the bronze-to-silver hop: rows that break it go to quarantine with a reason. Gold adds reconciliation (counts and sums agree with silver and the source).
Where does SCD Type 2 history live?
In silver, for entities whose history matters (a customer's city, a product's price). Gold dimensions read that history.
Does streaming need a different architecture?
Usually not: one medallion, two speeds. Hot tables flow through the same layers continuously; the rules (one-way flow, contracts, MERGE) stay the same.
Medallion vs Kimball vs data vault?
They answer different questions. Medallion is the flow and the promise of each layer. Kimball (star schemas) is how to shape gold for analysis. Data vault is one way to model an integrated, historised silver. Many platforms use medallion with Kimball in gold.
Next
Where to go next
Zoom out: Data platform architecture, drawn right: the control plane, the three services, streaming, 22 cross-cutting concerns and the agents around the medallion.
The ideas behind each layer, one short page each:
Learn it step by step: the Data Engineering path (Crawl, Walk, Run on AWS, Azure, Google Cloud, Databricks or open source).
Build one: the Ecom simple data platform project lands a shop's data through bronze, silver and gold. Join it, free.
Run it: DataTwin Studio follows the same three orders through every layer, recorded from a real Spark and Delta Lake run. Then send the same file again and watch a naive pipeline double the revenue.