cleandataengineering.aiData Platform, from the big picture to the code
You are here: Start here › Ingestion › Landing to bronze

Data PlatformZoom level 2

Landing to bronze: the lander, the bronze columns and the formats

This page is about the two ends of the lander: what a delivery in landing looks like, and what a row in bronze looks like once the lander has written it.

Both ends are built so that a mistake can be undone. Landing keeps the bytes as they arrived, so a bad read can be done again. Bronze keeps the batch key on every row, so a batch can be found, replaced or removed as a unit.

Delivery in landingdata file, thenits manifestPulled rowscopy, page,stream chunkOpen: Entry stepsEntry stepschecks, then stampzoom in ›Bronze writerreplace bybatch keyBronze tableone per datasetOpen: File logFile logone row per batchzoom in ›1 write2 record
The highlighted box is the port that writes. The bronze write always comes before the file log record.

Two landers in the code

The repository has two. src/dataplatform/lander.py is the first one, written for the first slice of the platform with a single dataset. It still runs in that slice's test. The lander that loads the nine tyre datasets is the function enter in src/dataplatform/ingest/steps.py, described in Entry steps. The two share ideas and differ in details, which the table lists.

First lander, lander.pyCurrent lander, ingest/steps.py
Reads fromLanding onlyLanding, pulled rows and stream chunks
Batch keyThe business date and the first 16 characters of the checksumThe first 32 characters of the checksum. The date is not part of it.
Columns stampedFourSix, plus transport columns for change feeds and streams
WriteAppendReplace by batch key
ChecksColumns and volumeColumns, types, volume, mass delete, tokens
src/dataplatform/lander.py:23The first lander's loop, for comparison. The shape is the same: skip what was handled, check, stamp, write, record.
def land(cfg: dict[str, Any], contract: dict[str, Any], landing: LandingStore, meta: MetadataStore,
         writer: TableWriter, runlog: RunLog, run_id: str, day: int) -> list[LandOutcome]:
    source, dataset = cfg["source_name"], cfg["dataset_name"]
    allowed_extra = {c for c, _ in CHANGE_COLUMNS}
    columns = odcs.columns(contract) + CHANGE_COLUMNS + LOAD_COLUMNS
    band = cfg.get("volume", {}).get("expected_rows_per_delivery", {})
    outcomes: list[LandOutcome] = []

    for d in sorted(landing.deliveries(source, dataset), key=lambda d: (d.window, d.name)):
        key = batch_key(d.window, d.checksum)
        file_id = f"file:{d.window}/{d.name}"
        if meta.batch_status(file_id) is not None:      # this file was handled by an earlier run
            continue
        if meta.batch_status(key) == "loaded":          # new file name, same window and bytes: a resend
            reason = f"same window and bytes as batch {key}"
            meta.record_batch(file_id, d, "skipped_duplicate", run_id, reason)
            outcomes.append(LandOutcome(d, key, "skipped_duplicate", 0, reason))
            continue
        rows = landing.read(d)
        observed = set().union(*(r.keys() for r in rows)) if rows else set()
        verdict = odcs.check_columns(contract, observed, allowed_extra)
        outcome = _decide(d, key, rows, verdict, band)
        if outcome.status == "loaded":
            stamped = [{**r, "_batch_key": key, "_run_id": run_id, "_source_file": d.name, "_window": d.window}
                       for r in rows]
            writer.append(cfg["bronze_table"], stamped, columns)
        if outcome.status == "loaded":
            meta.record_batch(key, d, "loaded", run_id)
        meta.record_batch(file_id, d, outcome.status, run_id, outcome.reason)
        outcomes.append(outcome)

Landing: a delivery is written once

src/dataplatform/ingest/ports.py:30The landing port. A mover writes through put. The lander reads through the other three.
class Landing(Protocol):
    """landing_store for push deliveries: the bytes exactly as delivered, with a manifest beside them."""

    def put(
        self,
        source: str,
        dataset: str,
        window: str,
        name: str,
        data: bytes,
        rows: int,
        extra: dict[str, Any] | None = None,
    ) -> Delivery:
        """Store a delivery and then its manifest. Raises DeliveryExists if (window, name) was landed before."""

    def deliveries(self, source: str, dataset: str) -> list[Delivery]:
        """Deliveries that have a manifest, in window then name order."""

    def read_bytes(self, delivery: Delivery) -> bytes: ...

    def manifest(self, delivery: Delivery) -> dict[str, Any]:
        """The manifest the mover wrote, including any extra fields (feed positions, arrival time)."""
src/dataplatform/adapters/files.py:37Landing on a plain folder. The data file is written first and its manifest last.
    def put(self, source: str, dataset: str, window: str, name: str, data: bytes, rows: int,
            extra: dict[str, Any] | None = None) -> Delivery:
        if "/" in name or name.startswith(".") or "/" in window:
            raise ValueError(f"not a plain delivery name: {window}/{name}")
        folder = self._dir(source, dataset) / window
        manifest_path = folder / f"{name}.manifest.json"
        if manifest_path.exists():
            raise DeliveryExists(f"{source}.{dataset} {window}/{name} is already landed; landing is written once")
        folder.mkdir(parents=True, exist_ok=True)
        (folder / name).write_bytes(data)
        manifest = {**(extra or {}), "source": source, "dataset": dataset, "window": window, "rows": rows,
                    "file": name}
        manifest_path.write_text(json.dumps(manifest, sort_keys=True))
        return Delivery(source, dataset, window, name, checksum(data), rows)

The layout is one folder per source, dataset and business date. A delivery is two files: the data exactly as delivered, and a small JSON manifest beside it with the row count and anything the mover wants to add. Two rules make landing safe. The manifest is written last, and a data file without a manifest is not a delivery, so a half-written file is never picked up. A second write of the same date and name is refused, so a replay always reads the bytes the first run read.

File formats

FormatSetting valueHow it is read
JSON linesjson_lines, also the fallbackOne JSON object per line. Empty lines are skipped.
Comma separatedcsvA header row is required. The separator is csv_delimiter.
Tab or pipe separatedtsv, pipeA header row is required.
ParquetparquetRead with a columnar library that must be installed. Types and missing values are kept.

Text formats give every value as text. The type step of the lander casts each value to the contract's type afterwards, so bronze has typed columns whatever the format was.

The columns stamped on every bronze row

src/dataplatform/ingest/steps.py:34The load columns, the transport columns by read type, and the two small sets used by the derive and protect steps.
LOAD_COLUMNS: Columns = [
    ("_batch_key", "string"),
    ("_run_id", "string"),
    ("_source_file", "string"),
    ("_window", "string"),
    ("_extra", "string"),
    ("_bronze_at", "timestamp"),  # when the lander wrote the row (ADR-034: every hop stamps its own time)
]
POSITION_COLUMNS: dict[str, Columns] = {  # where a row came from, by read type; bronze keeps them beside the row
    "change_feed": [("_op", "string"), ("_feed_pos", "integer"), ("_committed_at", "timestamp")],
    "events": [("_shard", "string"), ("_seq", "integer"), ("_arrived_at", "timestamp")],
}
DERIVED = frozenset({"file_name"})  # contract columns the lander fills from the delivery when the source does not
RESTRICTED = frozenset({"restricted"})  # personal-data classes tokenised before bronze
ColumnTypeWhat it holds
_batch_keystringThe key of the batch the row came in. Every row of a batch has the same one.
_run_idstringThe ingestion run that wrote the row.
_source_filestringThe batch's name: a file name, or a source position such as a copy time, a page range or a shard range.
_windowstringThe business date the batch belongs to.
_extrastringNew columns the contract does not have, as one JSON text. Empty when there are none.
_bronze_attimestampWhen the lander stamped the row.
Transport columnForWhat it holds
_opChange feedsThe operation: I for insert, U for update, D for delete.
_feed_posChange feedsThe row's position in the feed.
_committed_atChange feedsWhen the source committed the change.
_shardStreamsThe shard the record was read from.
_seqStreamsThe record's sequence number in its shard.
_arrived_atStreamsWhen the record arrived at the stream.
src/dataplatform/ingest/steps.py:78A bronze table's columns: the contract's, then the transport columns for its read type, then the load columns.
    @property
    def bronze_columns(self) -> Columns:
        return [(c.name, c.logical_type) for c in self.columns] + POSITION_COLUMNS.get(self.read, []) + LOAD_COLUMNS
src/dataplatform/silver/plan.py:70Silver's side of the same agreement. The two lists of names are written again here so silver knows which bronze columns are not business columns.
# Columns the lander stamps on every bronze row, and the transport columns some source types add.
LOAD_COLUMNS = ("_batch_key", "_run_id", "_source_file", "_window", "_extra", "_bronze_at")
TRANSPORT_COLUMNS = ("_op", "_feed_pos", "_committed_at", "_shard", "_seq", "_arrived_at")

The names are listed in two places, once in the ingestion steps and once in the silver plan. They agree today. Nothing in the code derives one from the other.

New columns are held in _extra

When a source starts sending a column the contract does not have, and the manifest allows it, the batch still loads. The contract columns go to their own bronze columns. The new ones are written together as one JSON text in _extra. Nothing is lost and the bronze table's shape does not change. The outcome carries a note naming the new columns, which reaches the file log as the reason. Promoting such a column into the contract is a change to the contract file.

The bronze writer and why a rerun changes nothing

src/dataplatform/ingest/ports.py:54The port. The first two methods write. The last two answer questions for reconciliation and the mass-delete guard.
class BronzeWriter(Protocol):
    """table_writer for bronze: append-only as delivered, idempotent by batch key."""

    def write(self, table: str, batch_key: str, rows: list[Row], columns: Columns, replace: bool = False) -> int:
        """Write one batch in one transaction, replacing whatever the table holds for this batch key: the same batch
        written twice leaves the table unchanged (DUP-01, DPC-0024) and a failed write leaves no trace (ORCH-03).
        `replace` is the lander saying "this batch may be in the table already" (a reload): a writer that appends new
        batches cheaply must then replace. Returns the rows written."""

    def write_many(self, table: str, batches: list[tuple[str, list[Row]]], columns: Columns) -> int:
        """Write several new batches of one table in one transaction (one commit instead of one per batch); all or
        nothing. The lander logs them as loaded only after this returns. Returns the rows written."""

    def fingerprint(
        self, table: str, where: dict[str, str] | None = None, sums: tuple[str, ...] = ()
    ) -> dict[str, float]:
        """{"row_count": n, <column>: total, ...} for the table, or for the rows where every column equals its value.
        A table never written has row_count 0."""

    def keys(self, table: str, columns: tuple[str, ...], where: dict[str, str]) -> set[tuple[Any, ...]]:
        """The distinct values of `columns` in the rows matching `where` (the mass-delete guard's comparison)."""

A write that can be repeated safely is called idempotent. The port states it as a rule: a write replaces whatever the table holds for its batch key. Since the batch key is a hash of the content, the same content written twice replaces itself with itself.

src/dataplatform/adapters/delta_bronze.py:56Bronze on Delta tables in a folder, without a cluster. One commit that overwrites only the rows of this batch key.
    def write(self, table: str, batch_key: str, rows: list[Row], columns: Columns, replace: bool = False) -> int:
        import pyarrow as pa
        from deltalake import write_deltalake

        check_batch_key(batch_key)  # it goes into the overwrite predicate
        schema = pa.schema([(c, _arrow_type(t)) for c, t in columns])
        data = pa.Table.from_pylist([{c: r.get(c) for c, _ in columns} for r in rows], schema=schema)
        handle = self._handle(table)
        write_deltalake(
            handle if handle is not None else str(self.path(table)),
            data,
            mode="overwrite",
            predicate=f"_batch_key = '{batch_key}'",
        )
        return int(data.num_rows)
src/dataplatform/adapters/delta_bronze.py:72Several new batches in one commit. The overwrite names exactly these batch keys.
    def write_many(self, table: str, batches: list[tuple[str, list[Row]]], columns: Columns) -> int:
        """New batches in one Delta commit; the lander logs them only after it returns."""
        import pyarrow as pa
        from deltalake import write_deltalake

        if not batches:
            return 0
        for k, _ in batches:
            check_batch_key(k)
        schema = pa.schema([(c, _arrow_type(t)) for c, t in columns])
        data = pa.Table.from_pylist([{c: r.get(c) for c, _ in columns} for _, part in batches for r in part],
                                    schema=schema)
        handle = self._handle(table)
        keys = ", ".join(f"'{k}'" for k, _ in batches)  # each checked by check_batch_key above
        # one commit that replaces exactly these batch keys: idempotent if a crash came between commit and file log
        write_deltalake(handle if handle is not None else str(self.path(table)), data, mode="overwrite",
                        predicate=f"_batch_key IN ({keys})")
        return int(data.num_rows)
src/dataplatform/ingest/ports.py:16A batch key ends up inside a filter text, so every writer checks its shape first.
SAFE_BATCH_KEY = re.compile(r"^[A-Za-z0-9_:.-]{1,128}$")  # batch keys reach a write predicate: every writer checks


def check_batch_key(key: str) -> str:
    if not SAFE_BATCH_KEY.match(key):
        raise ValueError(f"not a safe batch key: {key!r}")
    return key
AdapterFileWhere it writesUsed by
MemoryBronzesrc/dataplatform/adapters/memory_bronze.pyA list in memoryTests. It is the reference behaviour.
DeltaBronzesrc/dataplatform/adapters/delta_bronze.pyDelta tables in a local folder, through a library that needs no clusterThe local platform script and the 14-day test.
StoreBronzeWritersrc/dataplatform/adapters/table_store.pyDelta or Iceberg tables through the engine's table portThe engine binding.
SparkBronzeWritersrc/dataplatform/adapters/spark_delta.pyDelta tables in a catalogue through SparkThe port's contract test, when Spark is switched on for tests. I found no binding that uses it.
src/dataplatform/adapters/table_store.py:525The engine's writer. A new batch is appended with a write mark. Only a reload takes the replace path.
    def write(self, table: str, batch_key: str, rows: list[dict[str, Any]], columns: list[tuple[str, str]],
              replace: bool = False) -> int:
        from dataplatform.ingest.ports import check_batch_key  # noqa: PLC0415 -- no ingest import at load

        check_batch_key(batch_key)  # it goes into the replace predicate
        ddl = ", ".join(f"{quoted(c)} {self.TYPES[t]}" for c, t in columns)
        df = self.store.spark.createDataFrame([tuple(r.get(c) for c, _ in columns) for r in rows], ddl)
        name = self._table(table)
        # A new batch is appended (cheap; the batch key is the write's idempotency mark). Only a batch written before,
        # a reload, takes the replace path, which rewrites the rows of that batch key in one commit. Replacing on
        # every write scanned the whole bronze table per batch (13 Nov: most of a 3-day run's time).
        # Delta's write mark must rise, so Delta always takes its native replace (cheap and idempotent there)
        delta = self.store.format == "delta"
        if delta or replace or not self.store.append("bronze", name, df, f"bronze:{name}", batch_key):
            self.store.replace_where("bronze", name, df, f"_batch_key = '{batch_key}'")
        return len(rows)

The comment in this method records a measured lesson: replacing on every write scanned the whole bronze table per batch and took most of a run's time. So on one table format a new batch is appended, and the format's own write mark, made of an application name and the batch key, stops the same batch from being appended twice.

The file log entry

src/dataplatform/ingest/steps.py:379The record of a batch. It is written after the bronze commit.
def _record(ctx: Ctx, o: Outcome) -> Outcome:
    s = ctx.spec
    ctx.store.log_file(
        FileRecord(
            ctx.run_id,
            s.source,
            s.dataset,
            o.batch_key,
            o.batch.name,
            o.batch.window,
            hashlib.sha256(o.batch.data).hexdigest(),
            o.batch.declared_rows,
            o.rows,
            o.status,
            o.reason,
            "full" if s.read == "full_copy" else "incremental",  # a full copy is a rebase point (B8)
        )
    )
    return o

The file log is what the rest of the platform trusts. Silver decides what to read from it. A reload finds a window's batches in it. Reader: picking batches and Runtime tables show both.

A worked example: one bronze row

A change file for the tyre company's sales order lines arrives on 4 November 2026. The source has started to send a column named sales_channel that the contract does not have. The table shows one row as bronze holds it, with made-up values. The left column is the bronze column, the right column its value.

Bronze columnValue
order_id, line_noSO-1001, 10
promised_date, ordered_qty2026-11-12, 40
change_seq4321
_op, _feed_posU, 251
_batch_key32 hex characters, the same on every row of the file
_source_fileThe delivery's file name
_window2026-11-04
_extra{"sales_channel": "web"}

The settings this part reads

KeyMeaningDefault
file_format_data, file_encoding, csv_delimiter, csv_has_headerHow a delivered file is parsed.JSON lines, UTF-8, a comma, true
load.new_columnsWhether a new column is held in _extra or stops the batch.hold_in_bronze
ingest.readWhich transport columns the bronze table has.None for files, copies and pages
table_formatA proposed dataset setting for the table format. The engine binding chooses the format for the writer.No default in the dictionary

How it is tested

tests/dataplatform/test_ingest_ports.py is the contract test for the ports. Every bronze writer must pass the same table of cases, and every landing adapter must pass its own table. The cloud object storage landing runs against a local stand-in for the cloud service when that library is installed.

tests/dataplatform/test_ingest_ports.py:73The first cases every bronze writer must pass, among them the same batch written twice.
BRONZE_CASES = [
    ("a write returns its rows", write("k1", "AB"), 2),
    ("a table never written is empty", count(), 0),
    ("rows are kept", seq(write("k1", "AB"), count()), 2),
    ("the same batch twice changes nothing (DUP-01)", seq(write("k1", "AB"), write("k1", "AB"), count()), 2),
    ("a rewrite replaces only its batch", seq(write("k1", "AB"), write("k2", "C"), write("k1", "A"), count()), 2),
    ("batches side by side", seq(write("k1", "AB"), write("k2", "CDE"), count({"_batch_key": "k2"})), 3),
    ("filter on the run", seq(write("k1", "AB", "r1"), write("k2", "C", "r2"), count({"_run_id": "r2"})), 1),
    ("sums", seq(write("k1", "ABC"), lambda w: w.fingerprint("src.ds", None, ("qty",))["qty"]), 6),
tests/dataplatform/test_ingest_steps.py:288The stamp: after one file, every row has the run id, the file name, the batch key and the business date, and the restricted column holds a token.
def test_enter_stamps_and_protects():
    c = ctx()
    enter(Batch("2026-11-09", "f.csv", GOOD), c)
    rows = c.writer.tables["src.ds"]
    assert {r["_run_id"] for r in rows} == {"r1"} and {r["_source_file"] for r in rows} == {"f.csv"}
    assert {r["_batch_key"] for r in rows} == {content_key(GOOD)} and {r["_window"] for r in rows} == {"2026-11-09"}
    assert rows[0]["email"].startswith("tok_") and rows[1]["email"] == "" and rows[0]["qty"] == 1

tests/dataplatform/test_ingest_formats.py covers the delivered formats and the change feed column names.

Not built yet, or known gaps
  • SparkBronzeWriter.write_many appends without a replace and without a write mark. If a process died after that commit and before the file log was written, the next run would append the same batches again. The other three writers guard against this. I found this writer used only by the contract test.
  • The folder landing adapter reads every delivery's bytes to compute its checksum each time it lists the deliveries of a dataset. The cost grows with the number of files kept in landing.
  • The load column names are written in two modules that must be kept in step by hand.
  • A value in _extra is text inside JSON. It is not typed and silver does not read it.
  • Compressed and archive files are recognised by the discovery helper and cannot be parsed by the lander.
  • The first lander is still in the repository beside the current one. It is not on the path of the nine tyre datasets.