Data PlatformZoom level 2
The plan: a dataset's settings turned into one object
The stages never read the metadata store. They receive one object, the plan, that already holds every answer: the columns, the key, the kind of history, how duplicates are removed, how deletes are handled, and which rules apply.
Building the plan is also the gate. If a dataset's settings ask for something the engine cannot do, the plan refuses with a message that names the dataset and the setting, and no data is touched.
What the plan holds
src/dataplatform/silver/plan.py:133Every field of the plan. It is frozen: no stage can change it.@dataclass(frozen=True)
class SilverPlan:
source: str
dataset: str
columns: tuple[ColumnPlan, ...]
keys: tuple[str, ...]
history: str # scd1 | scd2 | append | snapshot | source_versioned
kind: str
ingest_read: str
dedup_mode: str = "none"
dedup_keys: tuple[str, ...] = ()
order_by: str = "" # history order (load.history_order_by), else dedup.order_by
track: tuple[str, ...] = () # columns that open a new version (SCD2); empty = every non-key column
effective_time: str = "" # the column that says when a change took effect (validity of versions)
event_time: str = "" # the event-time column (the watermark column when it is a timestamp): appended rows' time
delete_detect: str = "none"
delete_apply: str = "none"
snapshot_by: tuple[str, ...] = ()
valid_from: str = "" # source_versioned (ADR-032): the source's own validity columns
valid_to: str = ""
gaps_allowed: bool = False
lookback_days: int | None = None
projection: str = "all"
rules: tuple[Rule, ...] = ()
quality_actions: tuple[tuple[str, str], ...] = () # the dataset's own severity -> action overrides (ADR-009)
row_filters: tuple[Rule, ...] = ()
disabled: frozenset[str] = field(default_factory=frozenset)
token_pattern: str = ""
measures: tuple[Measure, ...] = ()| Field | Where it comes from | Which stage uses it |
|---|---|---|
columns | The contract's columns, each with the manifest's per-column settings laid over it | Stages 1 to 7 |
keys | The contract's primary key | Keys, history |
history, kind | history.type if given, else chosen from history.kind | History |
order_by | load.history_order_by, else dedup.order_by | Duplicates, history |
track | history.track_columns. Empty means every non-key column. | History, type 2 |
dedup_mode, dedup_keys | dedup.mode, dedup.business_keys | Duplicates |
delete_detect, delete_apply | load.deletes.detect, load.deletes.apply | History |
lookback_days | load.late_lookback_days | Duplicates, for appended events |
rules | Required columns, the quality actions, and the contract's row rules | Quality |
disabled | The processor_groups.* settings that say disabled | The stage runner |
token_pattern | The deployment's token format, passed in by the binding | Personal data |
History follows the kind of dataset
src/dataplatform/silver/plan.py:60The default history for each kind, and the five history types the engine knows.HISTORY_BY_KIND = {"master": "scd2", "transaction": "scd1", "event": "append", "snapshot": "snapshot"} # ADR-016
HISTORY_TYPES = ("scd1", "scd2", "append", "snapshot", "source_versioned") # ADR-016, ADR-032A dataset states its kind once: master, transaction, event or snapshot. The plan picks the history type from the kind. A dataset can override it. In the tyre example the sales order line is a transaction, which would normally keep only the latest state, but its manifest asks for type 2 so that every change to the promised date is kept.
data-platform-catalog/slices/02-tyre/manifest/datasets/erp.sales_order_line.yaml:25The sales order line's manifest: a transaction that asks for type 2 history on three columns.history: {kind: transaction, type: scd2, track_columns: [promised_date, ordered_qty, line_status]}
load:The checks, in the order the code makes them
src/dataplatform/silver/plan.py:253The first half of plan_from_view: every refusal.def plan_from_view(
view: dict[str, Any],
token_pattern: str = "",
projection: str = "all",
row_filters: tuple[Rule, ...] = (),
) -> SilverPlan:
"""The plan for one dataset from its merged view (MetadataStore.dataset_view). `token_pattern` is the binding's
tokenizer format; `projection` 'declared' keeps only the contract's columns (ADR-008: DPM has no key for it yet)."""
m = view["manifest"]
name = f"{view['source']}.{view['dataset']}"
overrides = {str(c["name"]): {k: v for k, v in c.items()} for c in (m.get("columns") or [])}
contract_names = {c["name"] for c in view["columns"]}
stray = sorted(set(overrides) - contract_names)
if stray:
raise PlanError(f"{name}: column settings for {', '.join(stray)}, which the contract does not have")
columns = tuple(_column(c, overrides.get(c["name"], {})) for c in view["columns"])
keys = tuple(view["business_keys"]["primary"])
if not keys:
raise PlanError(f"{name}: the contract names no primary key; silver needs one")
hist = m.get("history") or {}
kind = str(hist.get("kind") or view.get("history_kind") or "")
if kind not in HISTORY_BY_KIND:
raise PlanError(f"{name}: dataset kind {kind!r} is not one of {tuple(HISTORY_BY_KIND)} (ADR-016)")
history = str(hist.get("type") or HISTORY_BY_KIND[kind])
if history not in HISTORY_TYPES:
raise PlanError(f"{name}: history type {history!r} is not one of {HISTORY_TYPES}")
load, dedup = m.get("load") or {}, m.get("dedup") or {}
order_by = str(load.get("history_order_by") or dedup.get("order_by") or "")
if history in ("scd1", "scd2") and not order_by:
raise PlanError(f"{name}: {history} needs an order (load.history_order_by or dedup.order_by)")
names = {c.name for c in columns}
track = tuple(hist.get("track_columns") or ())
missing = sorted((set(track) | ({order_by} - {""})) - names)
if missing:
raise PlanError(f"{name}: history column(s) {', '.join(missing)} are not contract columns")
valid_from, valid_to, gaps = _validity(name, history, hist, view.get("settings") or {}, names)
deletes = load.get("deletes") or {}
if str(deletes.get("apply", "none")) not in DELETE_APPLY:
raise PlanError(f"{name}: deletes.apply {deletes.get('apply')!r} is not one of {DELETE_APPLY}")
mode = str(dedup.get("mode", "none"))
if mode not in DEDUP_MODES:
raise PlanError(f"{name}: dedup.mode {mode!r} is not one of {DEDUP_MODES}")
settings = view.get("settings") or {}
disabled = frozenset(s for s, group in SWITCHES.items() if settings.get(f"processor_groups.{group}") == "disabled")
if projection not in ("all", "declared"):- Stray column settings. The manifest has settings for a column the contract does not have. Refused.
- No primary key. The contract names no primary key. Silver cannot identify a row without one. Refused.
- Unknown kind or history type. The kind is not one of the four, or the history type is not one of the five. Refused.
- No order for keyed history. Type 1 and type 2 need an order column, because changes must be laid out in the source's order, not in arrival order. Refused if neither order setting is present.
- History columns not in the contract. A tracked column or the order column is not a contract column. Refused.
- Unknown delete or duplicate setting. The value is not in the allowed list. Refused.
- Stage switches. For each of the six optional stages, the plan reads its
processor_groupssetting. The worddisabledswitches the stage off.
src/dataplatform/silver/plan.py:48The six stages a dataset may switch off, and the setting that does it. The other seven always run.SWITCHES = {
"hygiene": "hygiene",
"parse_type": "type_safety",
"time": "timestamp_normalize",
"standardise": "standardization",
"personal_data": "pii",
"duplicates": "dedup",
}Two times the plan works out for you
The plan decides two time columns so that the stages do not have to guess. The effective time says when a change took effect, and gives versions their validity. For a change feed it is the commit time the feed carried. Otherwise it is the order column when that column is a timestamp. The event time is the time of an appended event. It is the watermark column when that is a timestamp, and otherwise the effective time.
How it is tested
tests/dataplatform/test_silver_plan.py is one table of cases. It builds the plan for every tyre dataset, checks each refusal with its message, checks the stage order, and checks that the silver kernel imports no engine.
- The dataset view does not carry the domain's quality defaults yet, so the plan falls back to defaults written beside it in the code.
- Keeping only the contract's columns (projection
declared) exists in the plan but has no setting in the manifest yet.