Data PlatformZoom level 2
The thirteen stages and the stage runner
Between bronze and silver every dataset passes the same thirteen stages in the same order. A stage is a plain function: it receives rows and the dataset's plan, and it returns rows. It never reads a table, never writes one and never looks at the clock.
Without a fixed list each table would grow its own cleaning job, and nobody could say what silver guarantees. With the list, one table of test cases per stage proves the behaviour once, and the only thing that differs between datasets is their settings.
The stages at a glance
Each line of this table is one stage. The fourth column says whether a dataset may switch the stage off. The last column says which rows the stage can set aside in the quarantine table. To quarantine a row means to take it out of the batch and store it with the reason, so that the good rows go on.
| No. | Stage | What it does | Can be switched off | What it can quarantine |
|---|---|---|---|---|
| 1 | Contract check | Checks that every contract column the plan keeps is present. A missing column stops the dataset run. | No | Nothing. It stops the whole run instead. |
| 2 | Project | Picks the columns, renames them to their contract names, extracts columns held in _extra, leaves out excluded columns. | No | Nothing. |
| 3 | Hygiene | Cleans text: byte-order mark, control characters, line endings, spaces at the ends, runs of spaces, empty text to a missing value. | Yes | Nothing. |
| 4 | Parse and type | Casts each column to the contract's type. A value that does not convert is never silently lost. | Yes | Rows with a value that does not convert, unless the column says to empty the value instead. |
| 5 | Time | Adds the local wall-clock time beside a timestamp when the column asks for it. | Yes | Nothing. |
| 6 | Standardise | Applies declared value maps and upper or lower case. | Yes | Nothing. |
| 7 | Personal data | Checks that every restricted column holds tokens, never the plain value. | Yes | Rows with a plain value in a restricted column. |
| 8 | Row filter | Drops rows that a declared filter does not want, and counts them. | No | Nothing. Dropped rows are counted, not stored. |
| 9 | Quality | Runs the row rules. The rule's severity decides whether a failing row is quarantined or kept with a flag. | No | Rows that break a rule whose action is quarantine. A critical failure also holds the batch. |
| 10 | Duplicates | Removes repeats inside the batch and, for appended events, rows already in silver. | Yes | Nothing. Dropped and replayed rows are counted. |
| 11 | Keys | Checks that every part of the business key is present and adds the key hash. | No | Rows with an empty key part. |
| 12 | History | Lays the rows out as versions, latest state, appended events or snapshots, by the kind of dataset, and marks deletes. | No | Only for datasets whose source keeps the versions itself: keys whose versions overlap or leave a gap. |
| 13 | Provenance | Stamps the silver run, the load time and a record hash, and puts the columns in a fixed order. | No | Nothing. |
What a stage is
Three small definitions in one file describe every stage. StageCtx is what a stage is told besides the rows. Out is what it gives back. Stage is the shape of the function itself.
src/dataplatform/stages/common.py:22What every stage receives besides the rows.@dataclass(frozen=True)
class StageCtx:
plan: SilverPlan
current: DataFrame | None # silver as it stands; None before the first write
run_id: str
as_of: str # the business date of the run (ISO)
loaded_at: datetime | None = None # when the write edge takes the rows (passed in: no clock inside a stage)| Field | What it is |
|---|---|
plan | The dataset's plan: columns, keys, history kind, rules. It is frozen, so no stage can change it. |
current | The silver table as it stands, handed over by the read edge. It is None before the first write. A stage may join with it. Only the duplicates and history stages do. |
run_id | The id of this silver run. The provenance stage stamps it on every row. |
as_of | The business date of the run, as text. The duplicates stage counts its lookback back from this date, not from today's date. |
loaded_at | The load time, read from the clock once by the adapter and passed in. A stage that read the clock itself would give different rows on every run and could not be tested with fixed answers. |
src/dataplatform/stages/common.py:31What every stage returns, and the type of a stage.@dataclass
class Out:
df: DataFrame
quarantine: DataFrame | None = None # QUARANTINE_COLUMNS
tally: dict[str, DataFrame] = field(default_factory=dict) # name -> rows to count (dropped, emptied, ...)
Stage = Callable[[DataFrame, StageCtx], Out]df is the rows that go on to the next stage. quarantine is the rows this stage set aside, or nothing. tally is a dictionary of named frames whose rows are to be counted, for example the rows a filter dropped. A stage hands over frames to count, not numbers, because counting runs a job, and a pure function should not run jobs. The adapter counts them all at once at the end.
The fixed order, and why this order
src/dataplatform/silver/plan.py:32The order is one tuple in the plan module. Nothing else decides it.STAGES = (
"contract_check",
"project",
"hygiene",
"parse_type",
"time",
"standardise",
"personal_data",
"row_filter",
"quality",
"duplicates",
"keys",
"history",
"provenance",
)src/dataplatform/stages/__init__.py:33The engine's functions by stage name. The last line refuses to import if the names or the order differ.FUNCTIONS: dict[str, Stage] = {
"contract_check": contract_check,
"project": project,
"hygiene": hygiene,
"parse_type": parse_type,
"time": time,
"standardise": standardise,
"personal_data": personal_data,
"row_filter": row_filter,
"quality": quality,
"duplicates": duplicates,
"keys": keys,
"history": history,
"provenance": provenance,
}
check_stage_order(list(FUNCTIONS)) # the engine supplies exactly the standard stages, in order (CFG-01)src/dataplatform/silver/plan.py:338The check that runs at import time.def check_stage_order(names: tuple[str, ...] | list[str]) -> None:
"""An engine must supply exactly the standard stages, in the standard order (DPC-0045, CFG-01)."""
if tuple(names) != STAGES:
raise PlanError(f"the stages must be exactly {STAGES} in this order; got {tuple(names)}")The order is checked when the module is imported, so a build with a stage missing, added or moved fails before any data is read. The order itself follows from what each stage needs from the ones before it. The table gives the reason for each position.
| Position | What the position gives |
|---|---|
| Contract check first | Every later stage addresses columns by name. A missing column must stop the run with a clear message, not fail somewhere in the middle with an engine error. |
| Project second | After it, every column carries its contract name, so all later stages and all rules can use contract names only. It also creates the _dq_flags column that later stages add to. |
| Hygiene before parse and type | The text ' 42 ' is trimmed before it is cast to a number, and empty text becomes a missing value before the cast, so neither counts as a failed cast. |
| Parse and type before time, standardise and quality | The time stage needs real timestamps. The rules in the quality stage compare typed values, for example a quantity with zero. |
| Standardise before quality | A rule that lists the valid values of a column sees the values after the value map and the letter case were applied. |
| Personal data, row filter and quality before duplicates | A row that is set aside or dropped takes no part in choosing which of two duplicates is kept. |
| Keys before history | History groups and joins on the key hash that the keys stage adds. |
| Provenance last | It stamps the run and fixes the column order after history has added its own columns. |
Which stages can be switched off, and how
src/dataplatform/silver/plan.py:48Six stages have a switch. The left side is the stage, the right side is the name of its setting group.SWITCHES = {
"hygiene": "hygiene",
"parse_type": "type_safety",
"time": "timestamp_normalize",
"standardise": "standardization",
"personal_data": "pii",
"duplicates": "dedup",
}A dataset switches a stage off with the setting processor_groups.<group> set to disabled. The group names differ from the stage names because the setting keys were kept from an earlier framework. The other seven stages have no switch and always run.
src/dataplatform/silver/plan.py:296How the plan turns the settings into the set of switched-off stages. disabled = frozenset(s for s, group in SWITCHES.items() if settings.get(f"processor_groups.{group}") == "disabled")src/dataplatform/silver/plan.py:177The one question the stage runner asks the plan. def runs(self, stage: str) -> bool:
return stage not in self.disabledThe comparison is with the exact word disabled. The settings registry also allows enabled, opt_in and dataset_decides for these keys, with no default. The plan treats all three, and a missing setting, the same way: the stage runs. How a setting's value is chosen between the default, the domain and the dataset is on the settings page.
| Stage | Setting that switches it off | What switching it off means |
|---|---|---|
| Hygiene | processor_groups.hygiene | Text reaches silver exactly as bronze has it. |
| Parse and type | processor_groups.type_safety | Columns keep the types bronze has. No cast failure is detected. |
| Time | processor_groups.timestamp_normalize | No local time column is added. |
| Standardise | processor_groups.standardization | Value maps and letter case are not applied. |
| Personal data | processor_groups.pii | Restricted columns are not checked for plain values. |
| Duplicates | processor_groups.dedup | No repeat is removed, and replays are not checked against silver. |
How the stages are chained
src/dataplatform/stages/__init__.py:53The whole stage runner. It is a loop over the fixed tuple.def run_stages(
plan: SilverPlan,
bronze: DataFrame,
current: DataFrame | None,
run_id: str,
as_of: str,
loaded_at: datetime | None = None,
hold: Callable[[DataFrame], DataFrame] | None = None,
) -> tuple[DataFrame, DataFrame | None, dict[str, DataFrame]]:
"""Every stage in order; a stage the plan switches off is skipped. Returns the rows to write, the quarantined
rows of every stage, and the frames to count, named <stage>.<what>. `hold`, when the engine adapter passes one,
fixes each stage's output before the next stage reads it (an engine that compiles a frame's whole lineage into
one query, as SQLFrame does, otherwise recompiles every earlier stage at every step; docs/DUCKDB-SPIKE.md)."""
ctx = StageCtx(plan, current, run_id, as_of, loaded_at)
df = bronze
held: list[DataFrame] = []
tally: dict[str, DataFrame] = {}
for name in STAGES:
if not plan.runs(name):
continue
out = FUNCTIONS[name](df, ctx)
df = out.df if hold is None else hold(out.df)
if out.quarantine is not None:
held.append(out.quarantine)
tally.update({f"{name}.{k}": v for k, v in out.tally.items()})
quarantine = reduce(lambda x, y: x.unionByName(y), held) if held else None
return df, quarantine, tally- Build the context. One
StageCtxis made from the plan, silver as it stands, the run id, the business date and the load time. Every stage receives the same one. - Start from bronze.
dfstarts as the new bronze rows. - Walk the tuple. For each stage name in
STAGES, the runner asksplan.runs(name). A switched-off stage is skipped anddfpasses through untouched. - Call the stage. The function is looked up by name and called with the rows and the context. Its
dfbecomes the input of the next stage. - Collect the quarantine. If the stage set rows aside, that frame is added to a list.
- Collect the counts. Each frame in the stage's
tallyis kept under the name<stage>.<what>, for exampleduplicates.dropped. - Return three things. The rows to write, one quarantine frame made by stacking the list, and the dictionary of frames to count.
Nothing has been computed when run_stages returns. On Spark a frame is a recipe, and the thirteen stages have only extended the recipe. The work happens later, when the adapter asks for the rows.
What a quarantine row holds
Four stages can set rows aside: parse and type, personal data, quality and keys. History can do it for one kind of dataset. They all use the same helper, so every quarantine row has the same five columns whatever stage made it. That is what lets the runner stack them with unionByName.
src/dataplatform/stages/common.py:41The helper that splits a frame into good rows and quarantine records.def quarantine_rows(
df: DataFrame, reason: Column, stage: str, record: list[str] | None = None
) -> tuple[DataFrame, DataFrame]:
"""Split a frame on `reason` (null = good row): (good rows, quarantine records keeping every original value of
the `record` columns, all columns by default)."""
marked = df.withColumn("_q_reason", reason)
good = marked.where(F.col("_q_reason").isNull()).drop("_q_reason")
data = list(record if record is not None else df.columns)
bad = marked.where(F.col("_q_reason").isNotNull()).select(
F.lit(stage).alias("_stage"),
F.col("_q_reason").alias("_reason"),
(F.col("_batch_key") if "_batch_key" in df.columns else F.lit(None).cast("string")).alias("_batch_key"),
(F.col("_source_file") if "_source_file" in df.columns else F.lit(None).cast("string")).alias("_source_file"),
F.to_json(F.struct(*[F.col(c) for c in data])).alias("_record"),
)
return good, badWalk through it. The stage passes a column expression called reason. It is text for a bad row and null for a good row. The helper adds it as a working column, keeps the rows where it is null as the good rows, and turns the others into records. Each record carries the stage name, the reason, the batch and the file the row came from, and the row itself as one JSON text. The record argument says which columns go into that JSON. By default it is all of them.
| Column | What it holds |
|---|---|
_stage | The stage that set the row aside, for example parse_type. |
_reason | Every check the row failed, in plain text. |
_batch_key | The bronze batch the row came from, or null if the frame has no such column. |
_source_file | The file the row came from, or null. |
_record | The whole row as JSON text, with the values as they were when the stage looked at them. |
src/dataplatform/stages/common.py:59How several failed checks on one row become one reason text.def reasons(parts: list[tuple[Column, str]]) -> Column:
"""One text naming every failed check, or null when none failed."""
if not parts:
return F.lit(None).cast("string")
texts = [F.when(cond, F.lit(text)) for cond, text in parts]
joined = F.concat_ws("; ", *texts)
return F.when(joined == "", F.lit(None)).otherwise(joined)A stage builds a list of pairs: a condition that is true when the check failed, and the text that names the failure. reasons turns each pair into a column that is the text when the condition holds and null otherwise. concat_ws joins the texts and skips the nulls, so a row that failed two checks gets both texts, and a row that failed none gets an empty text. The last line turns that empty text into null, which is the sign of a good row.
ExampleOne quarantine record for an order line whose quantity was sent as 4.5. The values are made up._stage parse_type
_reason ordered_qty is not a integer
_batch_key 2026-11-09T10:15|3f9a...
_source_file sales_order_line_1015.json
_record {"order_id":"SO-1002","line_no":"10","ordered_qty":"4.5", ... }The write edge appends these records to a table beside the silver table. The writer page shows that step.
How the counts are collected
The stage runner returns frames, not numbers. The adapter turns them into the counts that go to the run log. This table lists every count name the code can produce.
| Count | Where it comes from |
|---|---|
rows_in | The adapter counts the bronze rows it was given. |
rows_out, rows_removed | The adapter groups the final rows by their _action: upsert and delete. |
<stage>.quarantined | The adapter groups the quarantine records by their _stage column. |
parse_type.emptied | Rows where a value was emptied instead of quarantined. |
row_filter.dropped | Rows a declared filter dropped. |
quality.flagged, quality.critical | Rows kept with a flag, and rows that broke a critical rule. |
duplicates.dropped, duplicates.replayed | Repeats inside the batch, and rows already in silver. |
src/dataplatform/silver/runner.py:191After a successful write the runner logs one step per stage that has counts. for stage in STAGES:
mine = {k.split(".", 1)[1]: v for k, v in counts.items() if k.startswith(f"{stage}.")}
if mine:
status = "warned" if any(v for k, v in mine.items() if k in ("quarantined", "emptied")) else "succeeded"
store.log_step(StepRecord(run_id, source, dataset, f"silver.{stage}", status, mine))The runner cuts the stage name off each count and writes one step record per stage. The step's status is warned when the stage quarantined or emptied anything, and succeeded otherwise.
The hold argument
run_stages takes an optional function called hold. When it is given, the runner passes each stage's output through it before the next stage reads it. Its job is to fix the rows in place, so that the next stage starts from stored rows and not from a recipe.
Spark does not need it and passes nothing. The DuckDB adapter does need it. There, every frame is one SQL query, and each stage would make the query of all earlier stages longer. The adapter passes a function that runs the query once into a table.
src/dataplatform/adapters/duckdb_silver.py:67The DuckDB adapter passes its checkpoint function as hold. rows, quarantine, tally = run_stages(plan, bronze, current, run_id, as_of, datetime.now(UTC),
hold=checkpoint if self.hold_each_stage else None)The design note docs/DUCKDB-SPIKE.md records the effect: holding each stage's output halved the time of the stage tests on DuckDB, from 194 seconds to 92 seconds, and without any hold a history test over three batches did not finish in five minutes.
How the Spark adapter calls the stage runner
src/dataplatform/adapters/spark_silver.py:92The adapter between the silver runner and run_stages on Spark.class SparkStageRunner:
"""stage_runner on PySpark: the stages' frames are checkpointed once so the counts and the write read the same
rows (and a merge into the table the stages read from sees a fixed input)."""
def __init__(self, cache: bool = True):
self.cache = cache # serverless refuses cache() (PERSIST TABLE, probed 13 Nov): off there
def run(self, plan: SilverPlan, bronze: DataFrame, current: DataFrame | None, run_id: str,
as_of: str) -> StageResult:
if self.cache:
bronze = bronze.cache()
rows, quarantine, tally = run_stages(plan, bronze, current, run_id, as_of, datetime.now(UTC))
try:
rows = rows.localCheckpoint(eager=True)
except Exception as exc: # an engine error on this batch's data (e.g. a cast the plan did not foresee)
if self.cache:
bronze.unpersist()
raise StageError(f"{plan.table}: {type(exc).__name__}: {str(exc)[:500]}") from exc
counts = {"rows_in": bronze.count()}
by_action = {r["_action"]: r["n"] for r in rows.groupBy("_action").agg(F.count(F.lit(1)).alias("n")).collect()}
counts["rows_out"] = int(by_action.get("upsert", 0))
counts["rows_removed"] = int(by_action.get("delete", 0))
# everything else counted in one job: quarantine per stage and each tally (dropped, replayed, emptied, ...)
parts = [f.select(F.lit(name).alias("_what")) for name, f in tally.items()]
if quarantine is not None:
parts.append(quarantine.select(F.concat(F.col("_stage"), F.lit(".quarantined")).alias("_what")))
counts.update({name: 0 for name in tally})
if parts:
for r in reduce(lambda x, y: x.unionByName(y), parts).groupBy("_what").count().collect():
counts[str(r["_what"])] = int(r["count"])
if self.cache:
bronze.unpersist()
return StageResult(rows, quarantine, counts)- Cache bronze. The new bronze rows are cached, because they are read several times below. The cache can be turned off, because serverless compute refuses it.
- Read the clock once.
datetime.now(UTC)is passed as the load time. This is the only place the clock is read. - Run the recipe once.
localCheckpoint(eager=True)computes the final rows now and keeps them. The counts and the write then read the same rows. It also matters for the merge: the stages read the silver table, and a merge into that same table must see a fixed input. - Turn engine errors into a stage error. If Spark fails on the data, the adapter raises
StageErrorwith the first 500 characters of the message. The silver runner knows what to do with that error. - Count the rows. Bronze rows in, then the final rows grouped by action.
- Count everything else in one job. Each tally frame is reduced to one column holding its name. The quarantine frame is reduced to its stage name with
.quarantinedadded. All of them are stacked and grouped once. Every tally name starts at zero, so a count that found nothing is still reported. - Return. A
StageResultwith the rows, the quarantine frame and the counts.
src/dataplatform/silver/runner.py:172What the silver runner does with a StageError. except StageError as exc:
return _fail(store, run_id, view, "silver.stages", exc)A StageError from any stage fails the dataset run at the step silver.stages. Nothing is written and the bookmark does not move, so the next run takes the same batches again. The reader page explains the bookmark.
How it is tested
tests/dataplatform/test_silver_stages.py has one table of cases per stage, on one shared local Spark session. Each stage page shows cases from it. The adapter's own behaviour is in tests/dataplatform/test_silver_ports.py: its cases check the counts and that a batch which breaks its contract raises the stage error.
tests/dataplatform/test_silver_ports.py:149Three cases for the adapter's counts. ("rows in and out", APPEND, [("a", 1, 1, "I", ts(0)), ("b", 1, 1, "I", ts(0))],
{"rows_in": 2, "rows_out": 2, "rows_removed": 0}),
("duplicates counted", APPEND, [("a", 1, 1, "I", ts(0)), ("a", 1, 1, "I", ts(0))],
{"rows_in": 2, "rows_out": 1, "duplicates.dropped": 1}),
("quarantine counted per stage", {"required": ("v",), **APPEND},
[("a", None, 1, "I", ts(0)), ("b", 1, 1, "I", ts(0))], {"rows_out": 1, "quality.quarantined": 1}),The design decisions behind this page
| Decision | What it means | Why |
|---|---|---|
| Stages are pure functions | Frames in, frames out. No reads, no writes, no clock. | The same functions run on Spark and on DuckDB, and a test needs only a few rows in memory. |
| The order lives in one tuple | The kernel owns the list. An engine must supply exactly these names in this order. | An engine cannot quietly leave a stage out. |
| Switches are settings, not code | Six stages can be skipped per dataset by a registered setting. | A dataset never needs its own version of the pipeline. |
| Bad rows are set aside with a reason | A row that fails a check goes to quarantine. The good rows of the same batch go on. | One bad record must not stop a load, and nothing may disappear without a trace. |
- On Spark only the final rows are checkpointed. The quarantine frame is computed again when it is counted and again when it is written. The DuckDB adapter checkpoints both.
- There is no setting that adds a stage or changes the order. That is deliberate, but it means a dataset with a special need has to be handled before bronze or after silver.
- The settings registry also has the keys
processor_groups.provenanceandprocessor_groups.case_normalize. No code reads them, so setting them changes nothing. - Rows dropped by the row filter and the duplicates stage are counted but not stored anywhere.