Home › Learn › Data platform architecture › Ingestion
Data platform architecture · component · the same on every platform
Ingestion: sources to bronze
How data gets into a platform safely: six ingestion modes, mover vs lander, landing files, audit columns, batch checks and watermarks.
In plain words
Get everything in, exactly as it was
Ingestion has one job: copy data from the systems that create it into the platform, faithfully, without losing or doubling anything, and with a note on every row saying where and when it came from. It does not clean or reshape anything. That is silver's job.
It is split into two parts with one boundary between them:
The mover
Gets bytes from a source into the landing area as files. Swappable: Spark JDBC, Data Factory, Lakeflow Connect, dlt, a partner's SFTP. Holds no business logic.The lander
Our own code, the same on every platform. Reads landing files, adds the audit columns, appends to bronze in one commit, records the run. Owns watermarks, idempotency and schema drift.Because the lander is the same everywhere, changing the mover (say, from Data Factory to Lakeflow Connect) never changes what bronze looks like.
Choosing
Six ways to bring data in
| Mode | How it works | Good for | Watch out for |
|---|---|---|---|
| Full load | Copy the whole table every run | Small tables, reference data | Gets slower and dearer as data grows |
| Incremental (watermark) | Copy rows changed since the last high-water mark, e.g. updated_at > last value | Large tables with a reliable 'updated at' | Misses hard deletes; needs a column that really changes on every update |
| Change data capture (CDC) | Read the database's change log: every insert, update and delete in order | Big, busy tables; when deletes matter | Needs log access on the source; ordering and schema changes |
| File drop | Pick up new files from a folder or bucket | Partners, exports, legacy systems | Half-written files, duplicates, late files |
| API pull | Call an API page by page | SaaS apps | Rate limits, pagination, changing payloads |
| Event stream | Read a topic continuously | Clicks, devices, payments | Late and duplicate events; see the streaming lane |
The mode is a setting per dataset (ingestion.mode), not a different pipeline. See also full vs incremental and change data capture.
Metadata-driven
A new source is a few lines of YAML
To onboard the customers table, someone adds a dataset entry to the source's YAML (mode, key, history, checks, owner) and opens a pull request. The loader writes it into the metadata store; the generator produces the mover job, the lander job and the orchestration task from it. Nobody writes a new pipeline.
Landing
Files as received
- One folder per source, dataset and batch:
landing/shop_db/orders/batch=b-0912/. - Files are never edited and never deleted by a pipeline; a retention rule removes them after a few weeks.
- A batch is only picked up once a
_SUCCESSmarker (or the mover's own commit) says it is complete, so half-written files are never read. - Skipped when CDC writes straight into bronze.
The lander
Landing to bronze, safely
from pyspark.sql import functions as F
def land_to_bronze(spark, cfg: dict, batch_id: str, files: list[str], run_log):
if run_log.has_succeeded(cfg["dataset_id"], "bronze", batch_id):
return 0 # seen this batch: re-running adds nothing
df = (spark.read.format(cfg["landing.format"]) # files exactly as received
.options(**cfg.get("landing.read_options", {}))
.load(files))
df = (df.withColumn("_ingested_at", F.current_timestamp())
.withColumn("_source", F.lit(cfg["source_name"]))
.withColumn("_batch_id", F.lit(batch_id))
.withColumn("_file", F.col("_metadata.file_path")))
(df.write.format("delta")
.mode("append") # bronze is append-only
.option("mergeSchema", cfg.get("bronze.allow_new_columns", "true"))
.saveAsTable(cfg["bronze.table"]))
rows = df.count()
run_log.succeed(cfg["dataset_id"], "bronze", batch_id, rows_out=rows)
return rowsPySpark with Delta Lake. Settings come from the metadata store's merged config (key names illustrative).
Three things make it safe to run again and again:
- Batch check first. If the run log says this batch already reached bronze, it stops: a retried job adds nothing.
- One commit. The append is a single Delta transaction: readers see the whole batch or none of it.
- Watermark last. The watermark moves only after the run is recorded as successful.
1. read watermark -> 2026-10-26 09:00 2. mover copies rows with updated_at > watermark into landing/orders/batch=b-0912/ 3. lander appends landing files to bronze (one commit) 4. run_log: bronze b-0912 succeeded, 1,204 rows 5. ONLY NOW advance watermark -> max(updated_at) in the batch = 2026-10-27 08:59
The order of a watermark load. If the job dies after step 3, the watermark hasn't moved, the batch check stops a double append, and the next run picks up cleanly.
When the source changes shape
A new column is added to bronze automatically (bronze keeps the source's shape). A removed or renamed column, or a changed type, is not fixed in bronze: the data lands as it is, and the contract check at the bronze-to-silver hop stops it and alerts the owner.
What goes wrong
Common problems
The same batch lands twice
A retried job or a re-sent file doubles yesterday's rows in bronze.
Naive fix
DISTINCT in every query downstream.Right fix
Check the batch id against the run log before appending; dedupe by key again in silver.Deleted rows never disappear
A closed account still shows everywhere, because a watermark query only sees rows that still exist.
Naive fix
Reload everything every night.Right fix
CDC with delete events, or a periodic key comparison that flags missing keys as deleted in silver.The watermark skips rows
Rows updated in the same second as the last load are lost, or a row updated without touching updated_at never arrives.
Naive fix
Subtract an hour from the watermark and accept duplicates.Right fix
Use a column that always changes, overlap slightly and dedupe by key, or switch to CDC.A half-written file is read
Bronze gets a truncated batch.
Naive fix
Wait a few minutes before reading.Right fix
Only read a batch after its completion marker; the mover writes the marker last.The source goes dark for six hours
Alerts all night, then a flood.
Naive fix
Page someone every hour.Right fix
A freshness check with a sensible threshold, then a catch-up run; the batch check absorbs the flood without duplicates.Concept vs product
On each platform
| Job | AWS | Azure | Google Cloud | Databricks | Open source |
|---|---|---|---|---|---|
| Mover: databases | DMS, Glue JDBC | Data Factory copy, mirroring | Datastream | Lakeflow Connect, Spark JDBC | dlt, Airbyte, Spark JDBC |
| Mover: files and APIs | AppFlow, Transfer Family | Data Factory | Storage Transfer, Cloud Functions | Auto Loader | dlt |
| Landing storage | S3 | ADLS Gen2 / OneLake | Cloud Storage | Unity Catalog volumes | MinIO |
| Lander (our code) | Glue or EMR Spark | Fabric Spark | Dataproc | Databricks jobs | PySpark |
| Bronze tables | Iceberg on S3 | Delta in OneLake | Iceberg on BigLake | Delta in Unity Catalog | Delta or Iceberg |
Product names change often; check the current docs before you build.
FAQ
Questions people ask
Why not let the ingestion tool write bronze directly?
Then bronze looks different for every tool, and idempotency, audit columns and watermarks are re-implemented each time. A thin, shared lander keeps bronze identical whatever moved the data.
Should bronze have the source's types or all strings?
Keep the source's shape. Many teams land everything as strings to avoid failed loads; either way, typing is silver's job, where a failed cast goes to quarantine with a reason.
Do I need a landing area with CDC?
Usually not: CDC tools can write change events straight into bronze. Keep landing when you might need to re-parse the raw files.
How long should bronze be kept?
Long enough to rebuild silver after any bug you might find late, often months to years, within your privacy rules. Landing is short-lived.
Next
Where to go next
↑ The whole picture · The medallion layers · Key concepts · The Data Engineering path
See it run: DataTwin Studio plays this component step by step on a real recorded run, including what happens when the same file arrives twice.