Home › Learn › Data platform architecture › Ingestion

Data platform architecture · component · the same on every platform

Ingestion: sources to bronze

How data gets into a platform safely: six ingestion modes, mover vs lander, landing files, audit columns, batch checks and watermarks.

↑ The whole pictureMetadata storeIngestionBronze to silverGold

In plain words

Get everything in, exactly as it was

Ingestion has one job: copy data from the systems that create it into the platform, faithfully, without losing or doubling anything, and with a note on every row saying where and when it came from. It does not clean or reshape anything. That is silver's job.

Control plane: manifest → metadata DB → generators → pipelinesSourcesLandingBronzeSilverGoldSemanticExperiencesyou are here: Sources + Landing + Bronze
Ingestion covers sources, landing and bronze.

It is split into two parts with one boundary between them:

The mover

Gets bytes from a source into the landing area as files. Swappable: Spark JDBC, Data Factory, Lakeflow Connect, dlt, a partner's SFTP. Holds no business logic.

The lander

Our own code, the same on every platform. Reads landing files, adds the audit columns, appends to bronze in one commit, records the run. Owns watermarks, idempotency and schema drift.

Because the lander is the same everywhere, changing the mover (say, from Data Factory to Lakeflow Connect) never changes what bronze looks like.

Choosing

Six ways to bring data in

ModeHow it worksGood forWatch out for
Full loadCopy the whole table every runSmall tables, reference dataGets slower and dearer as data grows
Incremental (watermark)Copy rows changed since the last high-water mark, e.g. updated_at > last valueLarge tables with a reliable 'updated at'Misses hard deletes; needs a column that really changes on every update
Change data capture (CDC)Read the database's change log: every insert, update and delete in orderBig, busy tables; when deletes matterNeeds log access on the source; ordering and schema changes
File dropPick up new files from a folder or bucketPartners, exports, legacy systemsHalf-written files, duplicates, late files
API pullCall an API page by pageSaaS appsRate limits, pagination, changing payloads
Event streamRead a topic continuouslyClicks, devices, paymentsLate and duplicate events; see the streaming lane

The mode is a setting per dataset (ingestion.mode), not a different pipeline. See also full vs incremental and change data capture.

Metadata-driven

A new source is a few lines of YAML

To onboard the customers table, someone adds a dataset entry to the source's YAML (mode, key, history, checks, owner) and opens a pull request. The loader writes it into the metadata store; the generator produces the mover job, the lander job and the orchestration task from it. Nobody writes a new pipeline.

Landing

Files as received

The lander

Landing to bronze, safely

from pyspark.sql import functions as F

def land_to_bronze(spark, cfg: dict, batch_id: str, files: list[str], run_log):
    if run_log.has_succeeded(cfg["dataset_id"], "bronze", batch_id):
        return 0                                    # seen this batch: re-running adds nothing

    df = (spark.read.format(cfg["landing.format"])  # files exactly as received
               .options(**cfg.get("landing.read_options", {}))
               .load(files))

    df = (df.withColumn("_ingested_at", F.current_timestamp())
            .withColumn("_source",   F.lit(cfg["source_name"]))
            .withColumn("_batch_id", F.lit(batch_id))
            .withColumn("_file",     F.col("_metadata.file_path")))

    (df.write.format("delta")
       .mode("append")                              # bronze is append-only
       .option("mergeSchema", cfg.get("bronze.allow_new_columns", "true"))
       .saveAsTable(cfg["bronze.table"]))

    rows = df.count()
    run_log.succeed(cfg["dataset_id"], "bronze", batch_id, rows_out=rows)
    return rows

PySpark with Delta Lake. Settings come from the metadata store's merged config (key names illustrative).

Three things make it safe to run again and again:

  1. Batch check first. If the run log says this batch already reached bronze, it stops: a retried job adds nothing.
  2. One commit. The append is a single Delta transaction: readers see the whole batch or none of it.
  3. Watermark last. The watermark moves only after the run is recorded as successful.
1. read watermark            -> 2026-10-26 09:00
2. mover copies rows with updated_at > watermark into landing/orders/batch=b-0912/
3. lander appends landing files to bronze (one commit)
4. run_log: bronze b-0912 succeeded, 1,204 rows
5. ONLY NOW advance watermark -> max(updated_at) in the batch = 2026-10-27 08:59

The order of a watermark load. If the job dies after step 3, the watermark hasn't moved, the batch check stops a double append, and the next run picks up cleanly.

When the source changes shape

A new column is added to bronze automatically (bronze keeps the source's shape). A removed or renamed column, or a changed type, is not fixed in bronze: the data lands as it is, and the contract check at the bronze-to-silver hop stops it and alerts the owner.

What goes wrong

Common problems

The same batch lands twice

A retried job or a re-sent file doubles yesterday's rows in bronze.

Naive fix

DISTINCT in every query downstream.

Right fix

Check the batch id against the run log before appending; dedupe by key again in silver.

Deleted rows never disappear

A closed account still shows everywhere, because a watermark query only sees rows that still exist.

Naive fix

Reload everything every night.

Right fix

CDC with delete events, or a periodic key comparison that flags missing keys as deleted in silver.

The watermark skips rows

Rows updated in the same second as the last load are lost, or a row updated without touching updated_at never arrives.

Naive fix

Subtract an hour from the watermark and accept duplicates.

Right fix

Use a column that always changes, overlap slightly and dedupe by key, or switch to CDC.

A half-written file is read

Bronze gets a truncated batch.

Naive fix

Wait a few minutes before reading.

Right fix

Only read a batch after its completion marker; the mover writes the marker last.

The source goes dark for six hours

Alerts all night, then a flood.

Naive fix

Page someone every hour.

Right fix

A freshness check with a sensible threshold, then a catch-up run; the batch check absorbs the flood without duplicates.

Concept vs product

On each platform

JobAWSAzureGoogle CloudDatabricksOpen source
Mover: databasesDMS, Glue JDBCData Factory copy, mirroringDatastreamLakeflow Connect, Spark JDBCdlt, Airbyte, Spark JDBC
Mover: files and APIsAppFlow, Transfer FamilyData FactoryStorage Transfer, Cloud FunctionsAuto Loaderdlt
Landing storageS3ADLS Gen2 / OneLakeCloud StorageUnity Catalog volumesMinIO
Lander (our code)Glue or EMR SparkFabric SparkDataprocDatabricks jobsPySpark
Bronze tablesIceberg on S3Delta in OneLakeIceberg on BigLakeDelta in Unity CatalogDelta or Iceberg

Product names change often; check the current docs before you build.

FAQ

Questions people ask

Why not let the ingestion tool write bronze directly?

Then bronze looks different for every tool, and idempotency, audit columns and watermarks are re-implemented each time. A thin, shared lander keeps bronze identical whatever moved the data.

Should bronze have the source's types or all strings?

Keep the source's shape. Many teams land everything as strings to avoid failed loads; either way, typing is silver's job, where a failed cast goes to quarantine with a reason.

Do I need a landing area with CDC?

Usually not: CDC tools can write change events straight into bronze. Keep landing when you might need to re-parse the raw files.

How long should bronze be kept?

Long enough to rebuild silver after any bug you might find late, often months to years, within your privacy rules. Landing is short-lived.

Next

Where to go next

↑ The whole picture · The medallion layers · Key concepts · The Data Engineering path

See it run: DataTwin Studio plays this component step by step on a real recorded run, including what happens when the same file arrives twice.