Data PlatformZoom level 2
Landing to bronze: the lander, the bronze columns and the formats
This page is about the two ends of the lander: what a delivery in landing looks like, and what a row in bronze looks like once the lander has written it.
Both ends are built so that a mistake can be undone. Landing keeps the bytes as they arrived, so a bad read can be done again. Bronze keeps the batch key on every row, so a batch can be found, replaced or removed as a unit.
Two landers in the code
The repository has two. src/dataplatform/lander.py is the first one, written for the first slice of the platform with a single dataset. It still runs in that slice's test. The lander that loads the nine tyre datasets is the function enter in src/dataplatform/ingest/steps.py, described in Entry steps. The two share ideas and differ in details, which the table lists.
First lander, lander.py | Current lander, ingest/steps.py | |
|---|---|---|
| Reads from | Landing only | Landing, pulled rows and stream chunks |
| Batch key | The business date and the first 16 characters of the checksum | The first 32 characters of the checksum. The date is not part of it. |
| Columns stamped | Four | Six, plus transport columns for change feeds and streams |
| Write | Append | Replace by batch key |
| Checks | Columns and volume | Columns, types, volume, mass delete, tokens |
src/dataplatform/lander.py:23The first lander's loop, for comparison. The shape is the same: skip what was handled, check, stamp, write, record.def land(cfg: dict[str, Any], contract: dict[str, Any], landing: LandingStore, meta: MetadataStore,
writer: TableWriter, runlog: RunLog, run_id: str, day: int) -> list[LandOutcome]:
source, dataset = cfg["source_name"], cfg["dataset_name"]
allowed_extra = {c for c, _ in CHANGE_COLUMNS}
columns = odcs.columns(contract) + CHANGE_COLUMNS + LOAD_COLUMNS
band = cfg.get("volume", {}).get("expected_rows_per_delivery", {})
outcomes: list[LandOutcome] = []
for d in sorted(landing.deliveries(source, dataset), key=lambda d: (d.window, d.name)):
key = batch_key(d.window, d.checksum)
file_id = f"file:{d.window}/{d.name}"
if meta.batch_status(file_id) is not None: # this file was handled by an earlier run
continue
if meta.batch_status(key) == "loaded": # new file name, same window and bytes: a resend
reason = f"same window and bytes as batch {key}"
meta.record_batch(file_id, d, "skipped_duplicate", run_id, reason)
outcomes.append(LandOutcome(d, key, "skipped_duplicate", 0, reason))
continue
rows = landing.read(d)
observed = set().union(*(r.keys() for r in rows)) if rows else set()
verdict = odcs.check_columns(contract, observed, allowed_extra)
outcome = _decide(d, key, rows, verdict, band)
if outcome.status == "loaded":
stamped = [{**r, "_batch_key": key, "_run_id": run_id, "_source_file": d.name, "_window": d.window}
for r in rows]
writer.append(cfg["bronze_table"], stamped, columns)
if outcome.status == "loaded":
meta.record_batch(key, d, "loaded", run_id)
meta.record_batch(file_id, d, outcome.status, run_id, outcome.reason)
outcomes.append(outcome)Landing: a delivery is written once
src/dataplatform/ingest/ports.py:30The landing port. A mover writes through put. The lander reads through the other three.class Landing(Protocol):
"""landing_store for push deliveries: the bytes exactly as delivered, with a manifest beside them."""
def put(
self,
source: str,
dataset: str,
window: str,
name: str,
data: bytes,
rows: int,
extra: dict[str, Any] | None = None,
) -> Delivery:
"""Store a delivery and then its manifest. Raises DeliveryExists if (window, name) was landed before."""
def deliveries(self, source: str, dataset: str) -> list[Delivery]:
"""Deliveries that have a manifest, in window then name order."""
def read_bytes(self, delivery: Delivery) -> bytes: ...
def manifest(self, delivery: Delivery) -> dict[str, Any]:
"""The manifest the mover wrote, including any extra fields (feed positions, arrival time)."""src/dataplatform/adapters/files.py:37Landing on a plain folder. The data file is written first and its manifest last. def put(self, source: str, dataset: str, window: str, name: str, data: bytes, rows: int,
extra: dict[str, Any] | None = None) -> Delivery:
if "/" in name or name.startswith(".") or "/" in window:
raise ValueError(f"not a plain delivery name: {window}/{name}")
folder = self._dir(source, dataset) / window
manifest_path = folder / f"{name}.manifest.json"
if manifest_path.exists():
raise DeliveryExists(f"{source}.{dataset} {window}/{name} is already landed; landing is written once")
folder.mkdir(parents=True, exist_ok=True)
(folder / name).write_bytes(data)
manifest = {**(extra or {}), "source": source, "dataset": dataset, "window": window, "rows": rows,
"file": name}
manifest_path.write_text(json.dumps(manifest, sort_keys=True))
return Delivery(source, dataset, window, name, checksum(data), rows)The layout is one folder per source, dataset and business date. A delivery is two files: the data exactly as delivered, and a small JSON manifest beside it with the row count and anything the mover wants to add. Two rules make landing safe. The manifest is written last, and a data file without a manifest is not a delivery, so a half-written file is never picked up. A second write of the same date and name is refused, so a replay always reads the bytes the first run read.
File formats
| Format | Setting value | How it is read |
|---|---|---|
| JSON lines | json_lines, also the fallback | One JSON object per line. Empty lines are skipped. |
| Comma separated | csv | A header row is required. The separator is csv_delimiter. |
| Tab or pipe separated | tsv, pipe | A header row is required. |
| Parquet | parquet | Read with a columnar library that must be installed. Types and missing values are kept. |
Text formats give every value as text. The type step of the lander casts each value to the contract's type afterwards, so bronze has typed columns whatever the format was.
The columns stamped on every bronze row
src/dataplatform/ingest/steps.py:34The load columns, the transport columns by read type, and the two small sets used by the derive and protect steps.LOAD_COLUMNS: Columns = [
("_batch_key", "string"),
("_run_id", "string"),
("_source_file", "string"),
("_window", "string"),
("_extra", "string"),
("_bronze_at", "timestamp"), # when the lander wrote the row (ADR-034: every hop stamps its own time)
]
POSITION_COLUMNS: dict[str, Columns] = { # where a row came from, by read type; bronze keeps them beside the row
"change_feed": [("_op", "string"), ("_feed_pos", "integer"), ("_committed_at", "timestamp")],
"events": [("_shard", "string"), ("_seq", "integer"), ("_arrived_at", "timestamp")],
}
DERIVED = frozenset({"file_name"}) # contract columns the lander fills from the delivery when the source does not
RESTRICTED = frozenset({"restricted"}) # personal-data classes tokenised before bronze| Column | Type | What it holds |
|---|---|---|
_batch_key | string | The key of the batch the row came in. Every row of a batch has the same one. |
_run_id | string | The ingestion run that wrote the row. |
_source_file | string | The batch's name: a file name, or a source position such as a copy time, a page range or a shard range. |
_window | string | The business date the batch belongs to. |
_extra | string | New columns the contract does not have, as one JSON text. Empty when there are none. |
_bronze_at | timestamp | When the lander stamped the row. |
| Transport column | For | What it holds |
|---|---|---|
_op | Change feeds | The operation: I for insert, U for update, D for delete. |
_feed_pos | Change feeds | The row's position in the feed. |
_committed_at | Change feeds | When the source committed the change. |
_shard | Streams | The shard the record was read from. |
_seq | Streams | The record's sequence number in its shard. |
_arrived_at | Streams | When the record arrived at the stream. |
src/dataplatform/ingest/steps.py:78A bronze table's columns: the contract's, then the transport columns for its read type, then the load columns. @property
def bronze_columns(self) -> Columns:
return [(c.name, c.logical_type) for c in self.columns] + POSITION_COLUMNS.get(self.read, []) + LOAD_COLUMNSsrc/dataplatform/silver/plan.py:70Silver's side of the same agreement. The two lists of names are written again here so silver knows which bronze columns are not business columns.# Columns the lander stamps on every bronze row, and the transport columns some source types add.
LOAD_COLUMNS = ("_batch_key", "_run_id", "_source_file", "_window", "_extra", "_bronze_at")
TRANSPORT_COLUMNS = ("_op", "_feed_pos", "_committed_at", "_shard", "_seq", "_arrived_at")The names are listed in two places, once in the ingestion steps and once in the silver plan. They agree today. Nothing in the code derives one from the other.
New columns are held in _extra
When a source starts sending a column the contract does not have, and the manifest allows it, the batch still loads. The contract columns go to their own bronze columns. The new ones are written together as one JSON text in _extra. Nothing is lost and the bronze table's shape does not change. The outcome carries a note naming the new columns, which reaches the file log as the reason. Promoting such a column into the contract is a change to the contract file.
The bronze writer and why a rerun changes nothing
src/dataplatform/ingest/ports.py:54The port. The first two methods write. The last two answer questions for reconciliation and the mass-delete guard.class BronzeWriter(Protocol):
"""table_writer for bronze: append-only as delivered, idempotent by batch key."""
def write(self, table: str, batch_key: str, rows: list[Row], columns: Columns, replace: bool = False) -> int:
"""Write one batch in one transaction, replacing whatever the table holds for this batch key: the same batch
written twice leaves the table unchanged (DUP-01, DPC-0024) and a failed write leaves no trace (ORCH-03).
`replace` is the lander saying "this batch may be in the table already" (a reload): a writer that appends new
batches cheaply must then replace. Returns the rows written."""
def write_many(self, table: str, batches: list[tuple[str, list[Row]]], columns: Columns) -> int:
"""Write several new batches of one table in one transaction (one commit instead of one per batch); all or
nothing. The lander logs them as loaded only after this returns. Returns the rows written."""
def fingerprint(
self, table: str, where: dict[str, str] | None = None, sums: tuple[str, ...] = ()
) -> dict[str, float]:
"""{"row_count": n, <column>: total, ...} for the table, or for the rows where every column equals its value.
A table never written has row_count 0."""
def keys(self, table: str, columns: tuple[str, ...], where: dict[str, str]) -> set[tuple[Any, ...]]:
"""The distinct values of `columns` in the rows matching `where` (the mass-delete guard's comparison)."""A write that can be repeated safely is called idempotent. The port states it as a rule: a write replaces whatever the table holds for its batch key. Since the batch key is a hash of the content, the same content written twice replaces itself with itself.
src/dataplatform/adapters/delta_bronze.py:56Bronze on Delta tables in a folder, without a cluster. One commit that overwrites only the rows of this batch key. def write(self, table: str, batch_key: str, rows: list[Row], columns: Columns, replace: bool = False) -> int:
import pyarrow as pa
from deltalake import write_deltalake
check_batch_key(batch_key) # it goes into the overwrite predicate
schema = pa.schema([(c, _arrow_type(t)) for c, t in columns])
data = pa.Table.from_pylist([{c: r.get(c) for c, _ in columns} for r in rows], schema=schema)
handle = self._handle(table)
write_deltalake(
handle if handle is not None else str(self.path(table)),
data,
mode="overwrite",
predicate=f"_batch_key = '{batch_key}'",
)
return int(data.num_rows)src/dataplatform/adapters/delta_bronze.py:72Several new batches in one commit. The overwrite names exactly these batch keys. def write_many(self, table: str, batches: list[tuple[str, list[Row]]], columns: Columns) -> int:
"""New batches in one Delta commit; the lander logs them only after it returns."""
import pyarrow as pa
from deltalake import write_deltalake
if not batches:
return 0
for k, _ in batches:
check_batch_key(k)
schema = pa.schema([(c, _arrow_type(t)) for c, t in columns])
data = pa.Table.from_pylist([{c: r.get(c) for c, _ in columns} for _, part in batches for r in part],
schema=schema)
handle = self._handle(table)
keys = ", ".join(f"'{k}'" for k, _ in batches) # each checked by check_batch_key above
# one commit that replaces exactly these batch keys: idempotent if a crash came between commit and file log
write_deltalake(handle if handle is not None else str(self.path(table)), data, mode="overwrite",
predicate=f"_batch_key IN ({keys})")
return int(data.num_rows)src/dataplatform/ingest/ports.py:16A batch key ends up inside a filter text, so every writer checks its shape first.SAFE_BATCH_KEY = re.compile(r"^[A-Za-z0-9_:.-]{1,128}$") # batch keys reach a write predicate: every writer checks
def check_batch_key(key: str) -> str:
if not SAFE_BATCH_KEY.match(key):
raise ValueError(f"not a safe batch key: {key!r}")
return key| Adapter | File | Where it writes | Used by |
|---|---|---|---|
MemoryBronze | src/dataplatform/adapters/memory_bronze.py | A list in memory | Tests. It is the reference behaviour. |
DeltaBronze | src/dataplatform/adapters/delta_bronze.py | Delta tables in a local folder, through a library that needs no cluster | The local platform script and the 14-day test. |
StoreBronzeWriter | src/dataplatform/adapters/table_store.py | Delta or Iceberg tables through the engine's table port | The engine binding. |
SparkBronzeWriter | src/dataplatform/adapters/spark_delta.py | Delta tables in a catalogue through Spark | The port's contract test, when Spark is switched on for tests. I found no binding that uses it. |
src/dataplatform/adapters/table_store.py:525The engine's writer. A new batch is appended with a write mark. Only a reload takes the replace path. def write(self, table: str, batch_key: str, rows: list[dict[str, Any]], columns: list[tuple[str, str]],
replace: bool = False) -> int:
from dataplatform.ingest.ports import check_batch_key # noqa: PLC0415 -- no ingest import at load
check_batch_key(batch_key) # it goes into the replace predicate
ddl = ", ".join(f"{quoted(c)} {self.TYPES[t]}" for c, t in columns)
df = self.store.spark.createDataFrame([tuple(r.get(c) for c, _ in columns) for r in rows], ddl)
name = self._table(table)
# A new batch is appended (cheap; the batch key is the write's idempotency mark). Only a batch written before,
# a reload, takes the replace path, which rewrites the rows of that batch key in one commit. Replacing on
# every write scanned the whole bronze table per batch (13 Nov: most of a 3-day run's time).
# Delta's write mark must rise, so Delta always takes its native replace (cheap and idempotent there)
delta = self.store.format == "delta"
if delta or replace or not self.store.append("bronze", name, df, f"bronze:{name}", batch_key):
self.store.replace_where("bronze", name, df, f"_batch_key = '{batch_key}'")
return len(rows)The comment in this method records a measured lesson: replacing on every write scanned the whole bronze table per batch and took most of a run's time. So on one table format a new batch is appended, and the format's own write mark, made of an application name and the batch key, stops the same batch from being appended twice.
The file log entry
src/dataplatform/ingest/steps.py:379The record of a batch. It is written after the bronze commit.def _record(ctx: Ctx, o: Outcome) -> Outcome:
s = ctx.spec
ctx.store.log_file(
FileRecord(
ctx.run_id,
s.source,
s.dataset,
o.batch_key,
o.batch.name,
o.batch.window,
hashlib.sha256(o.batch.data).hexdigest(),
o.batch.declared_rows,
o.rows,
o.status,
o.reason,
"full" if s.read == "full_copy" else "incremental", # a full copy is a rebase point (B8)
)
)
return oThe file log is what the rest of the platform trusts. Silver decides what to read from it. A reload finds a window's batches in it. Reader: picking batches and Runtime tables show both.
A worked example: one bronze row
A change file for the tyre company's sales order lines arrives on 4 November 2026. The source has started to send a column named sales_channel that the contract does not have. The table shows one row as bronze holds it, with made-up values. The left column is the bronze column, the right column its value.
| Bronze column | Value |
|---|---|
order_id, line_no | SO-1001, 10 |
promised_date, ordered_qty | 2026-11-12, 40 |
change_seq | 4321 |
_op, _feed_pos | U, 251 |
_batch_key | 32 hex characters, the same on every row of the file |
_source_file | The delivery's file name |
_window | 2026-11-04 |
_extra | {"sales_channel": "web"} |
The settings this part reads
| Key | Meaning | Default |
|---|---|---|
file_format_data, file_encoding, csv_delimiter, csv_has_header | How a delivered file is parsed. | JSON lines, UTF-8, a comma, true |
load.new_columns | Whether a new column is held in _extra or stops the batch. | hold_in_bronze |
ingest.read | Which transport columns the bronze table has. | None for files, copies and pages |
table_format | A proposed dataset setting for the table format. The engine binding chooses the format for the writer. | No default in the dictionary |
How it is tested
tests/dataplatform/test_ingest_ports.py is the contract test for the ports. Every bronze writer must pass the same table of cases, and every landing adapter must pass its own table. The cloud object storage landing runs against a local stand-in for the cloud service when that library is installed.
tests/dataplatform/test_ingest_ports.py:73The first cases every bronze writer must pass, among them the same batch written twice.BRONZE_CASES = [
("a write returns its rows", write("k1", "AB"), 2),
("a table never written is empty", count(), 0),
("rows are kept", seq(write("k1", "AB"), count()), 2),
("the same batch twice changes nothing (DUP-01)", seq(write("k1", "AB"), write("k1", "AB"), count()), 2),
("a rewrite replaces only its batch", seq(write("k1", "AB"), write("k2", "C"), write("k1", "A"), count()), 2),
("batches side by side", seq(write("k1", "AB"), write("k2", "CDE"), count({"_batch_key": "k2"})), 3),
("filter on the run", seq(write("k1", "AB", "r1"), write("k2", "C", "r2"), count({"_run_id": "r2"})), 1),
("sums", seq(write("k1", "ABC"), lambda w: w.fingerprint("src.ds", None, ("qty",))["qty"]), 6),tests/dataplatform/test_ingest_steps.py:288The stamp: after one file, every row has the run id, the file name, the batch key and the business date, and the restricted column holds a token.def test_enter_stamps_and_protects():
c = ctx()
enter(Batch("2026-11-09", "f.csv", GOOD), c)
rows = c.writer.tables["src.ds"]
assert {r["_run_id"] for r in rows} == {"r1"} and {r["_source_file"] for r in rows} == {"f.csv"}
assert {r["_batch_key"] for r in rows} == {content_key(GOOD)} and {r["_window"] for r in rows} == {"2026-11-09"}
assert rows[0]["email"].startswith("tok_") and rows[1]["email"] == "" and rows[0]["qty"] == 1tests/dataplatform/test_ingest_formats.py covers the delivered formats and the change feed column names.
SparkBronzeWriter.write_manyappends without a replace and without a write mark. If a process died after that commit and before the file log was written, the next run would append the same batches again. The other three writers guard against this. I found this writer used only by the contract test.- The folder landing adapter reads every delivery's bytes to compute its checksum each time it lists the deliveries of a dataset. The cost grows with the number of files kept in landing.
- The load column names are written in two modules that must be kept in step by hand.
- A value in
_extrais text inside JSON. It is not typed and silver does not read it. - Compressed and archive files are recognised by the discovery helper and cannot be parsed by the lander.
- The first lander is still in the repository beside the current one. It is not on the path of the nine tyre datasets.