cleandataengineering.aiData Platform, from the big picture to the code
You are here: Start here › Bronze to silver › Reader: picking batches

Data PlatformZoom level 2

The reader: how the runner picks what to read from bronze

Silver must read each bronze batch exactly once, even after a crash, a rerun or a reload. It does not look at timestamps in the data to decide this. It asks the metadata store.

Two things in the metadata store make this work. The file log has one record for every batch ingestion handled, oldest first, with a status. The bookmark named silver.files is a number: how many file log records of this dataset silver has already passed.

record 1loadedrecord 2loadedrecord 3heldrecord 4loadedrecord 5voidrecord 6loadedbookmark = 2Records 1 and 2 were taken by an earlier run. This run looks at records 3 to 6and takes only the two that are loaded. The bookmark then moves to 6.
Each box is one record in the file log. The highlighted boxes are the batches this run reads.

The rules, one at a time

SituationWhat the reader doesWhy
A batch has status loadedTakes it.It is in bronze and passed ingestion's checks.
A batch was held or skipped by ingestionDoes not take it. The bookmark still moves past it.It never reached bronze, so there is nothing to read.
A batch was voidedNever takes it.Someone declared the batch bad. Voiding is how a bad batch is taken out.
The same batch key appears twiceTakes it once.A resent file has the same batch key. The stages would cope, but reading once is cheaper.
A batch has load type baselineSkips it in a normal run.A baseline batch is a compacted copy of what silver already holds, written by housekeeping before old bronze is removed. Only a rebuild reads it.
The dataset is being rebuiltStarts from record 0 and takes every loaded batch again.A rebuild drops the silver table and replays bronze, oldest first.
The dataset run failsThe bookmark does not move.The next run takes the same batches again, whole.
src/dataplatform/silver/runner.py:54The whole selection rule. It returns the batch keys to read and the new bookmark.
def new_batches(store: MetadataStore, source: str, dataset: str, from_start: bool = False) -> tuple[list[str], int]:
    """(batch keys loaded since the bookmark, in file-log order without repeats; the new bookmark). `from_start`
    takes every loaded batch from the first (a rebuild, ADR-025). A voided batch reads as `void` and is never taken.
    A baseline batch (housekeeping, B8) holds what silver already has: only a rebuild takes it."""
    files = store.dataset_files(source, dataset)
    start = 0 if from_start else int(store.bookmarks(source, dataset).get(BOOKMARK, "0"))
    keys: list[str] = []
    for f in files[start:]:
        if f.load_type == "baseline" and not from_start:
            continue
        if f.status == "loaded" and f.batch_key not in keys:
            keys.append(f.batch_key)
    return keys, len(files)

Walk through it. The function reads the dataset's file log. It starts at record 0 for a rebuild, otherwise at the bookmark. It then walks the remaining records in order. A baseline record is skipped unless this is a rebuild. A record is taken when its status is loaded and its batch key has not been taken already in this pass. The second value returned is the length of the file log, which becomes the new bookmark if the run succeeds.

Reading the rows

With the batch keys in hand, the read edge loads the bronze table and keeps the rows whose _batch_key is one of them. Ingestion stamped that column on every bronze row. The read edge also hands over the silver table as it stands, because the history stage needs to see what silver already holds.

src/dataplatform/adapters/spark_silver.py:160The read edge on Delta: new bronze rows by batch key, and silver as it stands.
    def bronze(self, plan: SilverPlan, batch_keys: list[str]) -> DataFrame:
        path = self.bronze_root / self._folder(plan.source, plan.dataset)
        df = self._read(path)
        if df is None:
            raise StageError(f"{plan.table}: no bronze table at {path}")
        return df.where(F.col("_batch_key").isin(batch_keys))

    def current(self, plan: SilverPlan) -> DataFrame | None:
        return self._read(self.silver_path(plan.source, plan.dataset))

Where the bookmark moves

src/dataplatform/silver/runner.py:157Nothing new, or only held batches: the two early exits.
    keys, mark = new_batches(store, source, dataset, from_start=rebuild)
    if not keys:
        start = int(store.bookmarks(source, dataset).get(BOOKMARK, "0"))
        if mark == start:  # nothing new in the file log
            store.log_step(StepRecord(run_id, source, dataset, "silver", "skipped", {}, "no new bronze batches"))
            store.finish_dataset_run(run_id, source, dataset, "skipped")
            return SilverDataset(source, dataset, "skipped")
        # only batches the lander held or skipped: nothing to take, and the bookmark moves past them (a bookmark
        # moves only with a succeeded dataset run)
        msg = f"{mark - start} new file record(s), none loaded: nothing to take"
        store.log_step(StepRecord(run_id, source, dataset, "silver", "succeeded", {"batches": 0}, msg))
        store.finish_dataset_run(run_id, source, dataset, "succeeded", {"batches": 0}, bookmarks={BOOKMARK: str(mark)})
        return SilverDataset(source, dataset, "succeeded", {"batches": 0}, reason=msg)

There are two early exits. If the file log has not grown, the dataset run is recorded as skipped and nothing else happens. If the file log grew but none of the new records is loaded, the run succeeds with zero batches and the bookmark moves past them. In every other case the bookmark is passed to finish_dataset_run at the very end, together with the success status. The metadata store saves the two in one transaction, so the bookmark can never move for a run that did not finish.

Why not read by a timestamp in bronze?

A timestamp bookmark misses a batch that is reloaded with an older time, and reads a batch twice when two runs overlap. The file log is written by ingestion in the same transaction that marks a batch loaded, so it is the one list that is always complete and in order.

How it is tested

tests/dataplatform/test_silver_runner.py runs the runner with in-memory fakes behind the ports. Its cases cover which batches are taken, the bookmark after success and after failure, the hold on a critical rule, and a rebuild.

Not built yet, or known gaps
  • The reader does not yet read bronze through a table change log. It trusts the file log and the batch key column.