cleandataengineering.aiData Platform, from the big picture to the code
You are here: Start here › Ingestion › Entry steps

Data PlatformZoom level 2

The entry steps every batch passes before bronze

The entry steps are the gate in front of bronze. Every batch, from any of the five pipelines, is handed to one function called enter, which runs ten steps in a fixed order and records the outcome.

The gate exists so that a broken delivery stops at the door. A file with a renamed column, a half-empty copy or a file sent twice should never reach the tables people query. And when a batch is turned away, the record must say which batch and why.

1 Seen beforebatch key2 Parsebytes to rows3 Columnsagainst contract4 Derivefile name5 Typescast values6 Volumerow counts7 Mass deletefull copies8 Protecttokens9 Stamp, writebronze10 Logfile logA batch that fails a step from 1 to 7 skips straight to step 10 with the reason.
Read the top row, then the bottom row. The highlighted box is the only step that writes data.

Goes in, what happens, comes out

Goes inOne batch: a business date, a name, the bytes, and for pulled batches the rows already parsed. Also the dataset's specification, built from the metadata store's view.
What happensTen steps. The first seven can each stop the batch. A stopped batch is not an error in the code. It is an outcome with a status and a reason.
Comes outOne outcome with one of five statuses. For a loaded batch, rows in bronze. In every case, one row in the file log.
StatusSet byMeaning
loadedStep 9The batch is in bronze.
skipped_duplicateStep 1The same bytes were loaded before, or were loaded and later voided.
rejected_contractSteps 2, 3 and 5The batch cannot be read, lacks a contract column, has a forbidden new column, or holds a value of the wrong type.
rejected_volumeStep 6The row count is not what was declared or is outside the expected band.
held_mass_deleteStep 7A full copy is missing too many keys.

The function that runs them

src/dataplatform/ingest/steps.py:315All ten steps in order. Each early return is one way a batch is turned away.
def enter(batch: Batch, ctx: Ctx) -> Outcome:
    """Run the steps for one batch and record the outcome. Never raises for a bad batch: it is held and logged."""
    spec, key = ctx.spec, content_key(batch.data)
    before = None if ctx.rewrite else ctx.store.batch_status(spec.source, spec.dataset, key)
    if before is None and ctx.pending is not None and any(o.batch_key == key and o.status == "loaded"
                                                          for o, _ in ctx.pending):
        before = "loaded"  # the same bytes earlier in this run, waiting for the commit
    if before == "loaded":
        return _log(ctx, Outcome(batch, key, "skipped_duplicate", 0, f"same bytes as batch {key}, already loaded"))
    if before == "void":  # ADR-025: a voided batch sent again stays out; only a reload may write it to bronze again
        return _log(ctx, Outcome(batch, key, "skipped_duplicate", 0, f"same bytes as batch {key}, which was voided"))
    try:
        # a copy: the steps change values in place and must never change what a source handed over
        rows = [dict(r) for r in batch.rows] if batch.rows is not None else parse(batch.data, spec.settings)
    except (ValueError, CastError) as exc:
        return _log(ctx, Outcome(batch, key, "rejected_contract", 0, f"cannot be read: {exc}"))
    reason, extra = check_columns(spec, rows)
    if reason:
        return _log(ctx, Outcome(batch, key, "rejected_contract", 0, reason))
    if "file_name" in {c.name for c in spec.columns}:
        for r in rows:
            r.setdefault("file_name", batch.name)
    reason = cast_rows([(c.name, c.logical_type) for c in spec.columns] + POSITION_COLUMNS.get(spec.read, []), rows)
    if reason:
        return _log(ctx, Outcome(batch, key, "rejected_contract", 0, reason))
    reason = volume_problem(spec, batch, len(rows))
    if reason:
        return _log(ctx, Outcome(batch, key, "rejected_volume", 0, reason))
    reason = mass_delete_problem(spec, rows, ctx.previous_keys, ctx.tokenizer)
    if reason:
        return _log(ctx, Outcome(batch, key, "held_mass_delete", 0, reason))
    tokens = protect(spec, rows, ctx.tokenizer)
    profile = profile_rows(rows, [(c.name, c.logical_type) for c in spec.columns])  # after tokenising (B9)
    stamped = stamp(spec, rows, key, ctx.run_id, batch, extra, ctx.now())
    notes = [f"new column(s) kept in bronze only: {', '.join(extra)} (SCH-01)"] if extra else []
    notes += [f"{tokens} personal value(s) tokenised"] if tokens else []
    if ctx.pending is not None and not ctx.rewrite:  # written with the run's other new batches in one commit
        o = Outcome(batch, key, "loaded", len(stamped), "; ".join(notes), notes, feed_marks(spec, rows), profile)
        ctx.pending.append((o, stamped))
        return o
    n = ctx.writer.write(spec.table, key, stamped, spec.bronze_columns, replace=ctx.rewrite)
    return _log(ctx, Outcome(batch, key, "loaded", n, "; ".join(notes), notes, feed_marks(spec, rows), profile))

The function never raises an error for a bad batch. Every exit goes through _log, so every batch leaves a record. The sections below take the steps one at a time.

The specification the steps work from

src/dataplatform/ingest/steps.py:82Everything the steps know about a dataset, taken from the store's view and never from a file.
    @classmethod
    def from_view(cls, view: dict[str, Any]) -> Spec:
        m = view["manifest"]
        band = (m.get("volume") or {}).get("expected_rows_per_delivery") or {}
        return cls(
            source=view["source"],
            dataset=view["dataset"],
            mode=view["ingest_mode"],
            read=view["ingest_read"],
            columns=tuple(Column(**c) for c in view["columns"]),
            primary_key=tuple(view["business_keys"]["primary"]),
            settings=dict(view["settings"]),
            load=dict(m.get("load") or {}),
            band=(int(band.get("min", 1)), band.get("max")),
            fingerprint=view["fingerprint"],
        )

The settings come from the merged settings of the view. The load block and the volume band come from the manifest JSON that the view carries. The band's lower bound is 1 when the manifest gives none.

Step 1: has this batch been seen before

src/dataplatform/ingest/steps.py:54The batch key: the first 32 hex characters of a hash of the bytes.
def content_key(data: bytes) -> str:
    """The batch key: the same bytes give the same key whatever the file is called or when it came (DUP-01)."""
    return hashlib.sha256(data).hexdigest()[:32]

The key depends only on the content. The first lines of enter ask the metadata store for the latest status of that key. If it is loaded the batch is skipped as a duplicate. If it is void, meaning a person withdrew the batch, it is skipped too, so a bad file sent again does not come back. The check also looks at batches waiting in the current run, in case the same bytes arrive twice in one run. A reload sets rewrite and passes this step on purpose.

For a pushed file the bytes are the file. For a pulled batch there is no file, so the pipeline passes the table name and the batch name as the bytes, for example the name of the copy or the page range. A pulled batch is therefore a duplicate when it has the same source position, not the same content.

Step 2: parse

src/dataplatform/ingest/steps.py:141Bytes to rows, by the dataset's file format setting.
def parse(data: bytes, settings: dict[str, str]) -> list[Row]:
    """A delivery's bytes as rows, by the dataset's file_format_data (ADR-031): json_lines (default), csv/tsv/pipe
    with a header, or parquet (change-capture and buffered-stream deliveries; types and nulls kept)."""
    fmt = settings.get("file_format_data", "json_lines")
    if fmt == "parquet":
        return _parquet_rows(data)
    text = data.decode(settings.get("file_encoding", "utf-8") or "utf-8")
    if fmt in ("csv", "tsv", "pipe"):
        delimiter = {"tsv": "\t", "pipe": "|"}.get(fmt, settings.get("csv_delimiter", ",") or ",")
        if settings.get("csv_has_header", "true") != "true":
            raise CastError("a CSV without a header row cannot be matched to the contract's columns")
        return [dict(r) for r in csv.DictReader(io.StringIO(text), delimiter=delimiter)]
    return [json.loads(line) for line in text.splitlines() if line.strip()]

Three families are read: JSON lines, which is the default, delimited text with a header row, and Parquet. A file that cannot be decoded or parsed makes the batch rejected_contract with the parser's message. Pulled batches arrive as rows and skip this step. The rows are copied first, because later steps change values in place and must not change what a source handed over.

Step 3: columns against the contract

src/dataplatform/ingest/steps.py:166Missing columns stop the batch. New columns stop it or are kept, as the manifest says.
def check_columns(spec: Spec, rows: list[Row]) -> tuple[str, list[str]]:
    """(reason the batch stops, or ''; new columns seen). An empty batch has no columns to judge: the volume step
    holds it."""
    if not rows:
        return "", []
    observed = {k for r in rows for k in r if not k.startswith("_")}
    names = {c.name for c in spec.columns}
    missing = sorted(names - observed - DERIVED)
    extra = sorted(observed - names)
    if missing:
        return (
            "missing contract column(s) "
            + ", ".join(missing)
            + " (renamed, removed or never sent)"
            + (f"; new column(s) seen: {', '.join(extra)}" if extra else "")
        ), extra
    if extra and spec.load.get("new_columns", "hold_in_bronze") == "stop":
        return f"new column(s) {', '.join(extra)} and the manifest says new_columns: stop", extra
    return "", extra

The observed columns are the keys of the rows, leaving out names that start with an underscore, which are the platform's own. A contract column that is not observed stops the batch, and the reason names it. One exception is a column the lander fills itself. A column the contract does not have is new. If the manifest says new_columns: stop the batch stops. Otherwise the new column is carried along and kept in bronze. Landing to bronze shows where it goes. Required columns are checked for presence only. A missing value in a required column is silver's to judge.

Step 4: derive

One contract column can be filled by the lander: file_name. The dealer sell-out contract has it, because the file a row came from decides which version of a month is the latest. If the contract has that column, every row that lacks it gets the batch's name. This is the three lines in enter that start with the test for file_name.

Step 5: types

src/dataplatform/ingest/steps.py:208One cast per contract type.
CASTS: dict[str, Callable[[Any], Any]] = {
    "string": str,
    "integer": _cast_int,
    "number": float,
    "timestamp": _cast_ts,
    "date": lambda v: v if isinstance(v, date) and not isinstance(v, datetime) else date.fromisoformat(str(v)),
    "boolean": _cast_bool,
}
src/dataplatform/ingest/steps.py:218Every value is cast in place. The first value that does not cast stops the whole batch.
def cast_rows(columns: Columns, rows: list[Row]) -> str:
    """Cast every value to its column's type, in place. Returns why the batch stops (naming the column), or ''.
    An empty text value is null for every type but string (CSV has no null)."""
    casts = [(name, CASTS[t], t == "string") for name, t in columns]
    for i, r in enumerate(rows):
        for name, cast, is_text in casts:
            v = r.get(name)
            if v is None or (v == "" and not is_text):
                r[name] = None
                continue
            try:
                r[name] = cast(v)
            except (ValueError, TypeError, CastError) as exc:
                return f"column {name} has {v!r} in row {i + 1}, which is not a {dict(columns)[name]} ({exc})"
    return ""

An empty text is treated as missing for every type except string, because CSV has no way to write a missing value. A timestamp without a time zone is taken as UTC. A number with a fraction is not accepted as an integer. The reason names the column, the value and the row, so the person who gets the alert knows where to look. For change feeds and streams, the transport columns are cast as well.

Step 6: volume

src/dataplatform/ingest/steps.py:235Two checks: the count the delivery declared, and the band the manifest expects.
def volume_problem(spec: Spec, batch: Batch, n: int) -> str:
    if batch.declared_rows is not None and n != batch.declared_rows:
        return f"the delivery manifest says {batch.declared_rows} rows, the batch has {n}"
    lo, hi = spec.band
    if n == 0 or n < lo or (hi is not None and n > int(hi)):
        return f"{n} rows is outside the expected {lo}-{hi if hi is not None else 'any'} (DPC-0013)"
    return ""

A push delivery has a manifest that states how many rows it holds. If the batch has a different number, the file was cut short or grew on the way. The second check uses the manifest's volume.expected_rows_per_delivery. An empty batch is always held, even when the band's minimum is zero.

Step 7: the mass-delete guard

src/dataplatform/ingest/steps.py:244A full copy that would delete more than the allowed share of keys is held.
def mass_delete_problem(spec: Spec, rows: list[Row], previous: set[tuple[Any, ...]] | None,
                        tokenizer: Tokenizer | None = None) -> str:
    """A full copy that would delete more than the declared share of the last accepted copy's keys (DEL-04).
    `previous` holds the keys as stored (a restricted key column holds its token), so the copy's keys are compared as
    stored too: with a `tokenizer`, a restricted key column is tokenised the way `protect` will."""
    limit = spec.load.get("mass_delete_max_share")
    if spec.read != "full_copy" or limit is None or not previous:
        return ""
    classes = {c.name: c.classification for c in spec.columns if c.classification in RESTRICTED}

    def stored(r: Row, k: str) -> Any:
        if tokenizer is not None and k in classes and r[k] not in (None, ""):
            return tokenizer.token(classes[k], str(r[k]))
        return r[k]

    now = {tuple(stored(r, k) for k in spec.primary_key) for r in rows}
    gone = len(previous - now) / len(previous)
    if gone > float(limit):
        return (
            f"mass delete held: {len(previous - now)} of {len(previous)} keys ({gone:.1%}) are missing, "
            f"more than the {float(limit):.0%} allowed (DEL-04); nothing appended, dataset blocked"
        )
    return ""

The guard applies only to full copies, only when the manifest sets load.mass_delete_max_share, and only when there is something to compare with. It builds the set of keys in the new copy, counts how many of the previous keys are gone, and holds the batch when that share is above the limit. Nothing is written. The previous keys are silver's current keys when silver exists, and otherwise the keys of the last accepted copy in bronze.

Step 8: protect personal data

src/dataplatform/ingest/steps.py:284Values of restricted columns are replaced by tokens before anything is written.
def protect(spec: Spec, rows: list[Row], tokenizer: Tokenizer) -> int:
    """Tokenise restricted columns in place; returns how many values were replaced."""
    restricted = [(c.name, c.classification) for c in spec.columns if c.classification in RESTRICTED]
    n = 0
    for r in rows:
        for name, cls in restricted:
            if r.get(name) not in (None, ""):
                r[name] = tokenizer.token(cls, str(r[name]))
                n += 1
    return n

A token is a stand-in value that cannot be turned back into the original without a secret key. A column is restricted when the contract classes it so. The real value never reaches bronze. The same value always gives the same token, so joins and duplicate checks still work. The batch's profile, a summary of its rows and columns, is taken after this step so it never sees a real value either.

src/dataplatform/adapters/keyed_tokens.py:24The tokenizer in use: a keyed hash, cut to 24 hex characters after a fixed prefix.
    def token(self, column_class: str, value: str) -> str:
        digest = hmac.new(self._key, f"{column_class}\x1f{value}".encode(), hashlib.sha256).hexdigest()
        return f"tok_{digest[:24]}"

Step 9: stamp and write

src/dataplatform/ingest/steps.py:296Each row gets the load columns. New columns are folded into one JSON text.
def stamp(spec: Spec, rows: list[Row], key: str, run_id: str, batch: Batch, extra: list[str],  # noqa: PLR0917
          at: datetime | None = None) -> list[Row]:
    names = [c for c, _ in spec.bronze_columns if c not in ("_extra",)]
    out = []
    for r in rows:
        row = {c: r.get(c) for c in names}
        row.update(
            _batch_key=key,
            _run_id=run_id,
            _source_file=batch.name,
            _window=batch.window,
            _extra=json.dumps({c: r[c] for c in extra if c in r}, sort_keys=True, default=str) if extra else None,
            _bronze_at=at,
        )
        out.append(row)
    return out

The row is rebuilt from the list of bronze columns, so its shape is the same for every batch of the dataset. Inside a normal run the stamped rows are not written at once. They are put on the pending list and written with the run's other new batches of the dataset.

src/dataplatform/ingest/steps.py:359One bronze commit for the run's new batches, then the file log.
def flush(ctx: Ctx) -> int:
    """Write the run's pending new batches in one commit, then log each as loaded: the file log never says loaded
    before bronze holds the rows. If the write fails nothing is logged and the next run takes the batches again."""
    if not ctx.pending:
        return 0
    new = [(o.batch_key, rows) for o, rows in ctx.pending if rows is not None]
    n = ctx.writer.write_many(ctx.spec.table, new, ctx.spec.bronze_columns) if new else 0
    for o, _ in ctx.pending:
        _record(ctx, o)
    ctx.pending.clear()
    return n

The order inside flush is the point. The write comes first. Only after it returns are the batches recorded. If the write fails, nothing is recorded and the next run finds the same batches again. If the process dies between the write and the record, the next run writes the same batch keys again, and the bronze writer replaces by batch key. Landing to bronze explains that.

Step 10: log

src/dataplatform/ingest/steps.py:379One file log row per batch, whatever the outcome.
def _record(ctx: Ctx, o: Outcome) -> Outcome:
    s = ctx.spec
    ctx.store.log_file(
        FileRecord(
            ctx.run_id,
            s.source,
            s.dataset,
            o.batch_key,
            o.batch.name,
            o.batch.window,
            hashlib.sha256(o.batch.data).hexdigest(),
            o.batch.declared_rows,
            o.rows,
            o.status,
            o.reason,
            "full" if s.read == "full_copy" else "incremental",  # a full copy is a rebase point (B8)
        )
    )
    return o

The row holds the batch key, the name, the business date, the full checksum, the declared row count, the rows loaded, the status and the reason. The load type is full for a full copy and incremental for everything else.

A worked example: three inspection files

The laboratory system sends CSV files of inspection results, separated by a character the manifest names. The manifest says new_columns: stop. Three files arrive. The file names are made up for this example. The first column is the file, the second what is in it, the third the outcome.

FileContentOutcome
inspection_P-CHN_1.csvTwo rows with every contract columnloaded, 2 rows in bronze
The same bytes again under the name inspection_P-CHN_1_resend.csvIdenticalskipped_duplicate, because the batch key is the same
inspection_P-HUN_1.csvThe column defect_code has been renamedrejected_contract. The reason names defect_code as missing and names the new column seen.

The dataset run fails because one batch was held. The two good outcomes stay as they are. The 14-day test plants this incident on day 7 and checks that the reason names the column and that the other plant's file keeps loading.

The settings the steps read

KeyStepMeaningAllowed values and default
file_format_dataParseThe format of the delivered file.The code reads json_lines, csv, tsv, pipe and parquet. The dictionary has no default. The code falls back to JSON lines.
file_encodingParseThe text encoding.Default utf-8.
csv_delimiterParseThe separator for csv.Default a comma.
csv_has_headerParseWhether the first line names the columns.Default true. A file without a header is refused.
load.new_columnsColumnsWhat to do with a column the contract does not have.hold_in_bronze or stop. The code's default is hold_in_bronze.
volume.expected_rows_per_deliveryVolumeThe smallest and largest row count expected.min defaults to 1. max may be absent.
load.mass_delete_max_shareMass deleteThe largest share of keys a full copy may lose.A fraction. No default, which switches the guard off. The two tyre master tables use 0.05.
The column's classificationProtectWhich columns are tokenised.restricted is the only class tokenised.

The first four are settings in the dictionary. The three with a dot in front come from blocks of the dataset manifest, which the steps read from the manifest JSON in the view. The last comes from the contract.

How it is tested

tests/dataplatform/test_ingest_steps.py has one table of cases per step, with no database and no engine. Each row is one input and what the step must say.

tests/dataplatform/test_ingest_steps.py:176The guard's cases: read type, previous keys, keys now, the limit, and whether the batch is held.
        ("first copy", "full_copy", None, ["A"], 0.05, False),
        ("nothing missing", "full_copy", {("A",), ("B",)}, ["A", "B", "C"], 0.05, False),
        ("40% missing", "full_copy", {(k,) for k in "ABCDEFGHIJ"}, list("ABCDEF"), 0.05, True),
        ("at the limit", "full_copy", {(k,) for k in "ABCDEFGHIJ"}, list("ABCDEFGHI"), 0.10, False),
        ("not a full copy", "files", {("A",), ("B",)}, [], 0.05, False),
        ("no limit declared", "full_copy", {("A",), ("B",)}, [], None, False),
tests/dataplatform/test_ingest_steps.py:273The whole gate: a list of batches, the statuses expected, and the rows that must be in bronze.
        ("a good file loads", [GOOD], ["loaded"], 2),
        ("the same bytes again are skipped (DUP-01)", [GOOD, GOOD], ["loaded", "skipped_duplicate"], 2),
        ("a renamed column is stopped (SCH-02)", [b"id;qty;remark;email\nA;1;;x\n"], ["rejected_contract"], 0),
        ("a retyped value is stopped (SCH-02)", [b"id;qty;note;email\nA;one;;x\n"], ["rejected_contract"], 0),
        ("an empty file is held (DPC-0013)", [b"id;qty;note;email\n"], ["rejected_volume"], 0),
tests/dataplatform/test_ingest_pipelines.py:305The guard through the runner: which keys it compares with in four situations.
        ("silver's keys are the comparison (DPC-0054): a short first copy is held", None, "ABCDEFGHIJ", "ABCDEF",
         "failed"),
        ("silver never written: the last accepted copy in bronze is the comparison", "ABCDEFGHIJ", None, "ABCDEF",
         "failed"),
        ("keys silver already marked deleted do not count as missing", "ABCDEFGHIJ", "ABCDEF", "ABCDEF",
         "succeeded"),
        ("a copy as long as silver loads", None, "ABCDEF", "ABCDEFG", "succeeded"),
Not built yet, or known gaps
  • The dictionary allows more file formats than the parser reads. Fixed width, XML, spreadsheets and Avro are allowed values of file_format_data, but any value the parser does not know is read as JSON lines and the batch is then rejected as unreadable.
  • The CSV keys for the quote character, the escape character and rows to skip are in the dictionary and are not read.
  • The manifest field load.batch_key lists what the batch key should be made of, and it differs between the tyre datasets. The code does not read it. The key is always the hash of the bytes the pipeline passes.
  • The manifest field load.quarantine_stop_share is not read by ingestion.
  • The volume check compares one batch with a fixed band. It does not compare with the dataset's own history. The profile step in the runner looks at drift and only warns.
  • A batch is all or nothing. One value of the wrong type stops every row of the file.
  • Each batch is held in memory as a list of rows. There is no streaming of a large file.