Data PlatformZoom level 2
The entry steps every batch passes before bronze
The entry steps are the gate in front of bronze. Every batch, from any of the five pipelines, is handed to one function called enter, which runs ten steps in a fixed order and records the outcome.
The gate exists so that a broken delivery stops at the door. A file with a renamed column, a half-empty copy or a file sent twice should never reach the tables people query. And when a batch is turned away, the record must say which batch and why.
Goes in, what happens, comes out
| Status | Set by | Meaning |
|---|---|---|
loaded | Step 9 | The batch is in bronze. |
skipped_duplicate | Step 1 | The same bytes were loaded before, or were loaded and later voided. |
rejected_contract | Steps 2, 3 and 5 | The batch cannot be read, lacks a contract column, has a forbidden new column, or holds a value of the wrong type. |
rejected_volume | Step 6 | The row count is not what was declared or is outside the expected band. |
held_mass_delete | Step 7 | A full copy is missing too many keys. |
The function that runs them
src/dataplatform/ingest/steps.py:315All ten steps in order. Each early return is one way a batch is turned away.def enter(batch: Batch, ctx: Ctx) -> Outcome:
"""Run the steps for one batch and record the outcome. Never raises for a bad batch: it is held and logged."""
spec, key = ctx.spec, content_key(batch.data)
before = None if ctx.rewrite else ctx.store.batch_status(spec.source, spec.dataset, key)
if before is None and ctx.pending is not None and any(o.batch_key == key and o.status == "loaded"
for o, _ in ctx.pending):
before = "loaded" # the same bytes earlier in this run, waiting for the commit
if before == "loaded":
return _log(ctx, Outcome(batch, key, "skipped_duplicate", 0, f"same bytes as batch {key}, already loaded"))
if before == "void": # ADR-025: a voided batch sent again stays out; only a reload may write it to bronze again
return _log(ctx, Outcome(batch, key, "skipped_duplicate", 0, f"same bytes as batch {key}, which was voided"))
try:
# a copy: the steps change values in place and must never change what a source handed over
rows = [dict(r) for r in batch.rows] if batch.rows is not None else parse(batch.data, spec.settings)
except (ValueError, CastError) as exc:
return _log(ctx, Outcome(batch, key, "rejected_contract", 0, f"cannot be read: {exc}"))
reason, extra = check_columns(spec, rows)
if reason:
return _log(ctx, Outcome(batch, key, "rejected_contract", 0, reason))
if "file_name" in {c.name for c in spec.columns}:
for r in rows:
r.setdefault("file_name", batch.name)
reason = cast_rows([(c.name, c.logical_type) for c in spec.columns] + POSITION_COLUMNS.get(spec.read, []), rows)
if reason:
return _log(ctx, Outcome(batch, key, "rejected_contract", 0, reason))
reason = volume_problem(spec, batch, len(rows))
if reason:
return _log(ctx, Outcome(batch, key, "rejected_volume", 0, reason))
reason = mass_delete_problem(spec, rows, ctx.previous_keys, ctx.tokenizer)
if reason:
return _log(ctx, Outcome(batch, key, "held_mass_delete", 0, reason))
tokens = protect(spec, rows, ctx.tokenizer)
profile = profile_rows(rows, [(c.name, c.logical_type) for c in spec.columns]) # after tokenising (B9)
stamped = stamp(spec, rows, key, ctx.run_id, batch, extra, ctx.now())
notes = [f"new column(s) kept in bronze only: {', '.join(extra)} (SCH-01)"] if extra else []
notes += [f"{tokens} personal value(s) tokenised"] if tokens else []
if ctx.pending is not None and not ctx.rewrite: # written with the run's other new batches in one commit
o = Outcome(batch, key, "loaded", len(stamped), "; ".join(notes), notes, feed_marks(spec, rows), profile)
ctx.pending.append((o, stamped))
return o
n = ctx.writer.write(spec.table, key, stamped, spec.bronze_columns, replace=ctx.rewrite)
return _log(ctx, Outcome(batch, key, "loaded", n, "; ".join(notes), notes, feed_marks(spec, rows), profile))The function never raises an error for a bad batch. Every exit goes through _log, so every batch leaves a record. The sections below take the steps one at a time.
The specification the steps work from
src/dataplatform/ingest/steps.py:82Everything the steps know about a dataset, taken from the store's view and never from a file. @classmethod
def from_view(cls, view: dict[str, Any]) -> Spec:
m = view["manifest"]
band = (m.get("volume") or {}).get("expected_rows_per_delivery") or {}
return cls(
source=view["source"],
dataset=view["dataset"],
mode=view["ingest_mode"],
read=view["ingest_read"],
columns=tuple(Column(**c) for c in view["columns"]),
primary_key=tuple(view["business_keys"]["primary"]),
settings=dict(view["settings"]),
load=dict(m.get("load") or {}),
band=(int(band.get("min", 1)), band.get("max")),
fingerprint=view["fingerprint"],
)The settings come from the merged settings of the view. The load block and the volume band come from the manifest JSON that the view carries. The band's lower bound is 1 when the manifest gives none.
Step 1: has this batch been seen before
src/dataplatform/ingest/steps.py:54The batch key: the first 32 hex characters of a hash of the bytes.def content_key(data: bytes) -> str:
"""The batch key: the same bytes give the same key whatever the file is called or when it came (DUP-01)."""
return hashlib.sha256(data).hexdigest()[:32]The key depends only on the content. The first lines of enter ask the metadata store for the latest status of that key. If it is loaded the batch is skipped as a duplicate. If it is void, meaning a person withdrew the batch, it is skipped too, so a bad file sent again does not come back. The check also looks at batches waiting in the current run, in case the same bytes arrive twice in one run. A reload sets rewrite and passes this step on purpose.
For a pushed file the bytes are the file. For a pulled batch there is no file, so the pipeline passes the table name and the batch name as the bytes, for example the name of the copy or the page range. A pulled batch is therefore a duplicate when it has the same source position, not the same content.
Step 2: parse
src/dataplatform/ingest/steps.py:141Bytes to rows, by the dataset's file format setting.def parse(data: bytes, settings: dict[str, str]) -> list[Row]:
"""A delivery's bytes as rows, by the dataset's file_format_data (ADR-031): json_lines (default), csv/tsv/pipe
with a header, or parquet (change-capture and buffered-stream deliveries; types and nulls kept)."""
fmt = settings.get("file_format_data", "json_lines")
if fmt == "parquet":
return _parquet_rows(data)
text = data.decode(settings.get("file_encoding", "utf-8") or "utf-8")
if fmt in ("csv", "tsv", "pipe"):
delimiter = {"tsv": "\t", "pipe": "|"}.get(fmt, settings.get("csv_delimiter", ",") or ",")
if settings.get("csv_has_header", "true") != "true":
raise CastError("a CSV without a header row cannot be matched to the contract's columns")
return [dict(r) for r in csv.DictReader(io.StringIO(text), delimiter=delimiter)]
return [json.loads(line) for line in text.splitlines() if line.strip()]Three families are read: JSON lines, which is the default, delimited text with a header row, and Parquet. A file that cannot be decoded or parsed makes the batch rejected_contract with the parser's message. Pulled batches arrive as rows and skip this step. The rows are copied first, because later steps change values in place and must not change what a source handed over.
Step 3: columns against the contract
src/dataplatform/ingest/steps.py:166Missing columns stop the batch. New columns stop it or are kept, as the manifest says.def check_columns(spec: Spec, rows: list[Row]) -> tuple[str, list[str]]:
"""(reason the batch stops, or ''; new columns seen). An empty batch has no columns to judge: the volume step
holds it."""
if not rows:
return "", []
observed = {k for r in rows for k in r if not k.startswith("_")}
names = {c.name for c in spec.columns}
missing = sorted(names - observed - DERIVED)
extra = sorted(observed - names)
if missing:
return (
"missing contract column(s) "
+ ", ".join(missing)
+ " (renamed, removed or never sent)"
+ (f"; new column(s) seen: {', '.join(extra)}" if extra else "")
), extra
if extra and spec.load.get("new_columns", "hold_in_bronze") == "stop":
return f"new column(s) {', '.join(extra)} and the manifest says new_columns: stop", extra
return "", extraThe observed columns are the keys of the rows, leaving out names that start with an underscore, which are the platform's own. A contract column that is not observed stops the batch, and the reason names it. One exception is a column the lander fills itself. A column the contract does not have is new. If the manifest says new_columns: stop the batch stops. Otherwise the new column is carried along and kept in bronze. Landing to bronze shows where it goes. Required columns are checked for presence only. A missing value in a required column is silver's to judge.
Step 4: derive
One contract column can be filled by the lander: file_name. The dealer sell-out contract has it, because the file a row came from decides which version of a month is the latest. If the contract has that column, every row that lacks it gets the batch's name. This is the three lines in enter that start with the test for file_name.
Step 5: types
src/dataplatform/ingest/steps.py:208One cast per contract type.CASTS: dict[str, Callable[[Any], Any]] = {
"string": str,
"integer": _cast_int,
"number": float,
"timestamp": _cast_ts,
"date": lambda v: v if isinstance(v, date) and not isinstance(v, datetime) else date.fromisoformat(str(v)),
"boolean": _cast_bool,
}src/dataplatform/ingest/steps.py:218Every value is cast in place. The first value that does not cast stops the whole batch.def cast_rows(columns: Columns, rows: list[Row]) -> str:
"""Cast every value to its column's type, in place. Returns why the batch stops (naming the column), or ''.
An empty text value is null for every type but string (CSV has no null)."""
casts = [(name, CASTS[t], t == "string") for name, t in columns]
for i, r in enumerate(rows):
for name, cast, is_text in casts:
v = r.get(name)
if v is None or (v == "" and not is_text):
r[name] = None
continue
try:
r[name] = cast(v)
except (ValueError, TypeError, CastError) as exc:
return f"column {name} has {v!r} in row {i + 1}, which is not a {dict(columns)[name]} ({exc})"
return ""An empty text is treated as missing for every type except string, because CSV has no way to write a missing value. A timestamp without a time zone is taken as UTC. A number with a fraction is not accepted as an integer. The reason names the column, the value and the row, so the person who gets the alert knows where to look. For change feeds and streams, the transport columns are cast as well.
Step 6: volume
src/dataplatform/ingest/steps.py:235Two checks: the count the delivery declared, and the band the manifest expects.def volume_problem(spec: Spec, batch: Batch, n: int) -> str:
if batch.declared_rows is not None and n != batch.declared_rows:
return f"the delivery manifest says {batch.declared_rows} rows, the batch has {n}"
lo, hi = spec.band
if n == 0 or n < lo or (hi is not None and n > int(hi)):
return f"{n} rows is outside the expected {lo}-{hi if hi is not None else 'any'} (DPC-0013)"
return ""A push delivery has a manifest that states how many rows it holds. If the batch has a different number, the file was cut short or grew on the way. The second check uses the manifest's volume.expected_rows_per_delivery. An empty batch is always held, even when the band's minimum is zero.
Step 7: the mass-delete guard
src/dataplatform/ingest/steps.py:244A full copy that would delete more than the allowed share of keys is held.def mass_delete_problem(spec: Spec, rows: list[Row], previous: set[tuple[Any, ...]] | None,
tokenizer: Tokenizer | None = None) -> str:
"""A full copy that would delete more than the declared share of the last accepted copy's keys (DEL-04).
`previous` holds the keys as stored (a restricted key column holds its token), so the copy's keys are compared as
stored too: with a `tokenizer`, a restricted key column is tokenised the way `protect` will."""
limit = spec.load.get("mass_delete_max_share")
if spec.read != "full_copy" or limit is None or not previous:
return ""
classes = {c.name: c.classification for c in spec.columns if c.classification in RESTRICTED}
def stored(r: Row, k: str) -> Any:
if tokenizer is not None and k in classes and r[k] not in (None, ""):
return tokenizer.token(classes[k], str(r[k]))
return r[k]
now = {tuple(stored(r, k) for k in spec.primary_key) for r in rows}
gone = len(previous - now) / len(previous)
if gone > float(limit):
return (
f"mass delete held: {len(previous - now)} of {len(previous)} keys ({gone:.1%}) are missing, "
f"more than the {float(limit):.0%} allowed (DEL-04); nothing appended, dataset blocked"
)
return ""The guard applies only to full copies, only when the manifest sets load.mass_delete_max_share, and only when there is something to compare with. It builds the set of keys in the new copy, counts how many of the previous keys are gone, and holds the batch when that share is above the limit. Nothing is written. The previous keys are silver's current keys when silver exists, and otherwise the keys of the last accepted copy in bronze.
Step 8: protect personal data
src/dataplatform/ingest/steps.py:284Values of restricted columns are replaced by tokens before anything is written.def protect(spec: Spec, rows: list[Row], tokenizer: Tokenizer) -> int:
"""Tokenise restricted columns in place; returns how many values were replaced."""
restricted = [(c.name, c.classification) for c in spec.columns if c.classification in RESTRICTED]
n = 0
for r in rows:
for name, cls in restricted:
if r.get(name) not in (None, ""):
r[name] = tokenizer.token(cls, str(r[name]))
n += 1
return nA token is a stand-in value that cannot be turned back into the original without a secret key. A column is restricted when the contract classes it so. The real value never reaches bronze. The same value always gives the same token, so joins and duplicate checks still work. The batch's profile, a summary of its rows and columns, is taken after this step so it never sees a real value either.
src/dataplatform/adapters/keyed_tokens.py:24The tokenizer in use: a keyed hash, cut to 24 hex characters after a fixed prefix. def token(self, column_class: str, value: str) -> str:
digest = hmac.new(self._key, f"{column_class}\x1f{value}".encode(), hashlib.sha256).hexdigest()
return f"tok_{digest[:24]}"Step 9: stamp and write
src/dataplatform/ingest/steps.py:296Each row gets the load columns. New columns are folded into one JSON text.def stamp(spec: Spec, rows: list[Row], key: str, run_id: str, batch: Batch, extra: list[str], # noqa: PLR0917
at: datetime | None = None) -> list[Row]:
names = [c for c, _ in spec.bronze_columns if c not in ("_extra",)]
out = []
for r in rows:
row = {c: r.get(c) for c in names}
row.update(
_batch_key=key,
_run_id=run_id,
_source_file=batch.name,
_window=batch.window,
_extra=json.dumps({c: r[c] for c in extra if c in r}, sort_keys=True, default=str) if extra else None,
_bronze_at=at,
)
out.append(row)
return outThe row is rebuilt from the list of bronze columns, so its shape is the same for every batch of the dataset. Inside a normal run the stamped rows are not written at once. They are put on the pending list and written with the run's other new batches of the dataset.
src/dataplatform/ingest/steps.py:359One bronze commit for the run's new batches, then the file log.def flush(ctx: Ctx) -> int:
"""Write the run's pending new batches in one commit, then log each as loaded: the file log never says loaded
before bronze holds the rows. If the write fails nothing is logged and the next run takes the batches again."""
if not ctx.pending:
return 0
new = [(o.batch_key, rows) for o, rows in ctx.pending if rows is not None]
n = ctx.writer.write_many(ctx.spec.table, new, ctx.spec.bronze_columns) if new else 0
for o, _ in ctx.pending:
_record(ctx, o)
ctx.pending.clear()
return nThe order inside flush is the point. The write comes first. Only after it returns are the batches recorded. If the write fails, nothing is recorded and the next run finds the same batches again. If the process dies between the write and the record, the next run writes the same batch keys again, and the bronze writer replaces by batch key. Landing to bronze explains that.
Step 10: log
src/dataplatform/ingest/steps.py:379One file log row per batch, whatever the outcome.def _record(ctx: Ctx, o: Outcome) -> Outcome:
s = ctx.spec
ctx.store.log_file(
FileRecord(
ctx.run_id,
s.source,
s.dataset,
o.batch_key,
o.batch.name,
o.batch.window,
hashlib.sha256(o.batch.data).hexdigest(),
o.batch.declared_rows,
o.rows,
o.status,
o.reason,
"full" if s.read == "full_copy" else "incremental", # a full copy is a rebase point (B8)
)
)
return oThe row holds the batch key, the name, the business date, the full checksum, the declared row count, the rows loaded, the status and the reason. The load type is full for a full copy and incremental for everything else.
A worked example: three inspection files
The laboratory system sends CSV files of inspection results, separated by a character the manifest names. The manifest says new_columns: stop. Three files arrive. The file names are made up for this example. The first column is the file, the second what is in it, the third the outcome.
| File | Content | Outcome |
|---|---|---|
inspection_P-CHN_1.csv | Two rows with every contract column | loaded, 2 rows in bronze |
The same bytes again under the name inspection_P-CHN_1_resend.csv | Identical | skipped_duplicate, because the batch key is the same |
inspection_P-HUN_1.csv | The column defect_code has been renamed | rejected_contract. The reason names defect_code as missing and names the new column seen. |
The dataset run fails because one batch was held. The two good outcomes stay as they are. The 14-day test plants this incident on day 7 and checks that the reason names the column and that the other plant's file keeps loading.
The settings the steps read
| Key | Step | Meaning | Allowed values and default |
|---|---|---|---|
file_format_data | Parse | The format of the delivered file. | The code reads json_lines, csv, tsv, pipe and parquet. The dictionary has no default. The code falls back to JSON lines. |
file_encoding | Parse | The text encoding. | Default utf-8. |
csv_delimiter | Parse | The separator for csv. | Default a comma. |
csv_has_header | Parse | Whether the first line names the columns. | Default true. A file without a header is refused. |
load.new_columns | Columns | What to do with a column the contract does not have. | hold_in_bronze or stop. The code's default is hold_in_bronze. |
volume.expected_rows_per_delivery | Volume | The smallest and largest row count expected. | min defaults to 1. max may be absent. |
load.mass_delete_max_share | Mass delete | The largest share of keys a full copy may lose. | A fraction. No default, which switches the guard off. The two tyre master tables use 0.05. |
The column's classification | Protect | Which columns are tokenised. | restricted is the only class tokenised. |
The first four are settings in the dictionary. The three with a dot in front come from blocks of the dataset manifest, which the steps read from the manifest JSON in the view. The last comes from the contract.
How it is tested
tests/dataplatform/test_ingest_steps.py has one table of cases per step, with no database and no engine. Each row is one input and what the step must say.
tests/dataplatform/test_ingest_steps.py:176The guard's cases: read type, previous keys, keys now, the limit, and whether the batch is held. ("first copy", "full_copy", None, ["A"], 0.05, False),
("nothing missing", "full_copy", {("A",), ("B",)}, ["A", "B", "C"], 0.05, False),
("40% missing", "full_copy", {(k,) for k in "ABCDEFGHIJ"}, list("ABCDEF"), 0.05, True),
("at the limit", "full_copy", {(k,) for k in "ABCDEFGHIJ"}, list("ABCDEFGHI"), 0.10, False),
("not a full copy", "files", {("A",), ("B",)}, [], 0.05, False),
("no limit declared", "full_copy", {("A",), ("B",)}, [], None, False),tests/dataplatform/test_ingest_steps.py:273The whole gate: a list of batches, the statuses expected, and the rows that must be in bronze. ("a good file loads", [GOOD], ["loaded"], 2),
("the same bytes again are skipped (DUP-01)", [GOOD, GOOD], ["loaded", "skipped_duplicate"], 2),
("a renamed column is stopped (SCH-02)", [b"id;qty;remark;email\nA;1;;x\n"], ["rejected_contract"], 0),
("a retyped value is stopped (SCH-02)", [b"id;qty;note;email\nA;one;;x\n"], ["rejected_contract"], 0),
("an empty file is held (DPC-0013)", [b"id;qty;note;email\n"], ["rejected_volume"], 0),tests/dataplatform/test_ingest_pipelines.py:305The guard through the runner: which keys it compares with in four situations. ("silver's keys are the comparison (DPC-0054): a short first copy is held", None, "ABCDEFGHIJ", "ABCDEF",
"failed"),
("silver never written: the last accepted copy in bronze is the comparison", "ABCDEFGHIJ", None, "ABCDEF",
"failed"),
("keys silver already marked deleted do not count as missing", "ABCDEFGHIJ", "ABCDEF", "ABCDEF",
"succeeded"),
("a copy as long as silver loads", None, "ABCDEF", "ABCDEFG", "succeeded"),- The dictionary allows more file formats than the parser reads. Fixed width, XML, spreadsheets and Avro are allowed values of
file_format_data, but any value the parser does not know is read as JSON lines and the batch is then rejected as unreadable. - The CSV keys for the quote character, the escape character and rows to skip are in the dictionary and are not read.
- The manifest field
load.batch_keylists what the batch key should be made of, and it differs between the tyre datasets. The code does not read it. The key is always the hash of the bytes the pipeline passes. - The manifest field
load.quarantine_stop_shareis not read by ingestion. - The volume check compares one batch with a fixed band. It does not compare with the dataset's own history. The profile step in the runner looks at drift and only warns.
- A batch is all or nothing. One value of the wrong type stops every row of the file.
- Each batch is held in memory as a list of rows. There is no streaming of a large file.