Data PlatformZoom level 2
The reader: how the runner picks what to read from bronze
Silver must read each bronze batch exactly once, even after a crash, a rerun or a reload. It does not look at timestamps in the data to decide this. It asks the metadata store.
Two things in the metadata store make this work. The file log has one record for every batch ingestion handled, oldest first, with a status. The bookmark named silver.files is a number: how many file log records of this dataset silver has already passed.
The rules, one at a time
| Situation | What the reader does | Why |
|---|---|---|
A batch has status loaded | Takes it. | It is in bronze and passed ingestion's checks. |
| A batch was held or skipped by ingestion | Does not take it. The bookmark still moves past it. | It never reached bronze, so there is nothing to read. |
| A batch was voided | Never takes it. | Someone declared the batch bad. Voiding is how a bad batch is taken out. |
| The same batch key appears twice | Takes it once. | A resent file has the same batch key. The stages would cope, but reading once is cheaper. |
A batch has load type baseline | Skips it in a normal run. | A baseline batch is a compacted copy of what silver already holds, written by housekeeping before old bronze is removed. Only a rebuild reads it. |
| The dataset is being rebuilt | Starts from record 0 and takes every loaded batch again. | A rebuild drops the silver table and replays bronze, oldest first. |
| The dataset run fails | The bookmark does not move. | The next run takes the same batches again, whole. |
src/dataplatform/silver/runner.py:54The whole selection rule. It returns the batch keys to read and the new bookmark.def new_batches(store: MetadataStore, source: str, dataset: str, from_start: bool = False) -> tuple[list[str], int]:
"""(batch keys loaded since the bookmark, in file-log order without repeats; the new bookmark). `from_start`
takes every loaded batch from the first (a rebuild, ADR-025). A voided batch reads as `void` and is never taken.
A baseline batch (housekeeping, B8) holds what silver already has: only a rebuild takes it."""
files = store.dataset_files(source, dataset)
start = 0 if from_start else int(store.bookmarks(source, dataset).get(BOOKMARK, "0"))
keys: list[str] = []
for f in files[start:]:
if f.load_type == "baseline" and not from_start:
continue
if f.status == "loaded" and f.batch_key not in keys:
keys.append(f.batch_key)
return keys, len(files)Walk through it. The function reads the dataset's file log. It starts at record 0 for a rebuild, otherwise at the bookmark. It then walks the remaining records in order. A baseline record is skipped unless this is a rebuild. A record is taken when its status is loaded and its batch key has not been taken already in this pass. The second value returned is the length of the file log, which becomes the new bookmark if the run succeeds.
Reading the rows
With the batch keys in hand, the read edge loads the bronze table and keeps the rows whose _batch_key is one of them. Ingestion stamped that column on every bronze row. The read edge also hands over the silver table as it stands, because the history stage needs to see what silver already holds.
src/dataplatform/adapters/spark_silver.py:160The read edge on Delta: new bronze rows by batch key, and silver as it stands. def bronze(self, plan: SilverPlan, batch_keys: list[str]) -> DataFrame:
path = self.bronze_root / self._folder(plan.source, plan.dataset)
df = self._read(path)
if df is None:
raise StageError(f"{plan.table}: no bronze table at {path}")
return df.where(F.col("_batch_key").isin(batch_keys))
def current(self, plan: SilverPlan) -> DataFrame | None:
return self._read(self.silver_path(plan.source, plan.dataset))Where the bookmark moves
src/dataplatform/silver/runner.py:157Nothing new, or only held batches: the two early exits. keys, mark = new_batches(store, source, dataset, from_start=rebuild)
if not keys:
start = int(store.bookmarks(source, dataset).get(BOOKMARK, "0"))
if mark == start: # nothing new in the file log
store.log_step(StepRecord(run_id, source, dataset, "silver", "skipped", {}, "no new bronze batches"))
store.finish_dataset_run(run_id, source, dataset, "skipped")
return SilverDataset(source, dataset, "skipped")
# only batches the lander held or skipped: nothing to take, and the bookmark moves past them (a bookmark
# moves only with a succeeded dataset run)
msg = f"{mark - start} new file record(s), none loaded: nothing to take"
store.log_step(StepRecord(run_id, source, dataset, "silver", "succeeded", {"batches": 0}, msg))
store.finish_dataset_run(run_id, source, dataset, "succeeded", {"batches": 0}, bookmarks={BOOKMARK: str(mark)})
return SilverDataset(source, dataset, "succeeded", {"batches": 0}, reason=msg)There are two early exits. If the file log has not grown, the dataset run is recorded as skipped and nothing else happens. If the file log grew but none of the new records is loaded, the run succeeds with zero batches and the bookmark moves past them. In every other case the bookmark is passed to finish_dataset_run at the very end, together with the success status. The metadata store saves the two in one transaction, so the bookmark can never move for a run that did not finish.
A timestamp bookmark misses a batch that is reloaded with an older time, and reads a batch twice when two runs overlap. The file log is written by ingestion in the same transaction that marks a batch loaded, so it is the one list that is always complete and in order.
How it is tested
tests/dataplatform/test_silver_runner.py runs the runner with in-memory fakes behind the ports. Its cases cover which batches are taken, the bookmark after success and after failure, the hold on a critical rule, and a rebuild.
- The reader does not yet read bronze through a table change log. It trusts the file log and the batch key column.