cleandataengineering.aiData Platform, from the big picture to the code
You are here: Start here › Bronze to silver › The plan

Data PlatformZoom level 2

The plan: a dataset's settings turned into one object

The stages never read the metadata store. They receive one object, the plan, that already holds every answer: the columns, the key, the kind of history, how duplicates are removed, how deletes are handled, and which rules apply.

Building the plan is also the gate. If a dataset's settings ask for something the engine cannot do, the plan refuses with a message that names the dataset and the setting, and no data is touched.

Dataset manifesthistory, load, dedupContractcolumns, types, keyOpen: Merged settingsMerged settingsdefault, domain, datasetzoom in ›Open: dataset_viewdataset_viewone JSON documentzoom in ›plan_from_viewchecks, then buildsSilverPlanfrozen
The metadata store merges the three sources into one document. The plan is built from that document alone.

What the plan holds

src/dataplatform/silver/plan.py:133Every field of the plan. It is frozen: no stage can change it.
@dataclass(frozen=True)
class SilverPlan:
    source: str
    dataset: str
    columns: tuple[ColumnPlan, ...]
    keys: tuple[str, ...]
    history: str  # scd1 | scd2 | append | snapshot | source_versioned
    kind: str
    ingest_read: str
    dedup_mode: str = "none"
    dedup_keys: tuple[str, ...] = ()
    order_by: str = ""  # history order (load.history_order_by), else dedup.order_by
    track: tuple[str, ...] = ()  # columns that open a new version (SCD2); empty = every non-key column
    effective_time: str = ""  # the column that says when a change took effect (validity of versions)
    event_time: str = ""  # the event-time column (the watermark column when it is a timestamp): appended rows' time
    delete_detect: str = "none"
    delete_apply: str = "none"
    snapshot_by: tuple[str, ...] = ()
    valid_from: str = ""  # source_versioned (ADR-032): the source's own validity columns
    valid_to: str = ""
    gaps_allowed: bool = False
    lookback_days: int | None = None
    projection: str = "all"
    rules: tuple[Rule, ...] = ()
    quality_actions: tuple[tuple[str, str], ...] = ()  # the dataset's own severity -> action overrides (ADR-009)
    row_filters: tuple[Rule, ...] = ()
    disabled: frozenset[str] = field(default_factory=frozenset)
    token_pattern: str = ""
    measures: tuple[Measure, ...] = ()
FieldWhere it comes fromWhich stage uses it
columnsThe contract's columns, each with the manifest's per-column settings laid over itStages 1 to 7
keysThe contract's primary keyKeys, history
history, kindhistory.type if given, else chosen from history.kindHistory
order_byload.history_order_by, else dedup.order_byDuplicates, history
trackhistory.track_columns. Empty means every non-key column.History, type 2
dedup_mode, dedup_keysdedup.mode, dedup.business_keysDuplicates
delete_detect, delete_applyload.deletes.detect, load.deletes.applyHistory
lookback_daysload.late_lookback_daysDuplicates, for appended events
rulesRequired columns, the quality actions, and the contract's row rulesQuality
disabledThe processor_groups.* settings that say disabledThe stage runner
token_patternThe deployment's token format, passed in by the bindingPersonal data

History follows the kind of dataset

src/dataplatform/silver/plan.py:60The default history for each kind, and the five history types the engine knows.
HISTORY_BY_KIND = {"master": "scd2", "transaction": "scd1", "event": "append", "snapshot": "snapshot"}  # ADR-016
HISTORY_TYPES = ("scd1", "scd2", "append", "snapshot", "source_versioned")  # ADR-016, ADR-032

A dataset states its kind once: master, transaction, event or snapshot. The plan picks the history type from the kind. A dataset can override it. In the tyre example the sales order line is a transaction, which would normally keep only the latest state, but its manifest asks for type 2 so that every change to the promised date is kept.

data-platform-catalog/slices/02-tyre/manifest/datasets/erp.sales_order_line.yaml:25The sales order line's manifest: a transaction that asks for type 2 history on three columns.
history: {kind: transaction, type: scd2, track_columns: [promised_date, ordered_qty, line_status]}
load:

The checks, in the order the code makes them

src/dataplatform/silver/plan.py:253The first half of plan_from_view: every refusal.
def plan_from_view(
    view: dict[str, Any],
    token_pattern: str = "",
    projection: str = "all",
    row_filters: tuple[Rule, ...] = (),
) -> SilverPlan:
    """The plan for one dataset from its merged view (MetadataStore.dataset_view). `token_pattern` is the binding's
    tokenizer format; `projection` 'declared' keeps only the contract's columns (ADR-008: DPM has no key for it yet)."""
    m = view["manifest"]
    name = f"{view['source']}.{view['dataset']}"
    overrides = {str(c["name"]): {k: v for k, v in c.items()} for c in (m.get("columns") or [])}
    contract_names = {c["name"] for c in view["columns"]}
    stray = sorted(set(overrides) - contract_names)
    if stray:
        raise PlanError(f"{name}: column settings for {', '.join(stray)}, which the contract does not have")
    columns = tuple(_column(c, overrides.get(c["name"], {})) for c in view["columns"])
    keys = tuple(view["business_keys"]["primary"])
    if not keys:
        raise PlanError(f"{name}: the contract names no primary key; silver needs one")
    hist = m.get("history") or {}
    kind = str(hist.get("kind") or view.get("history_kind") or "")
    if kind not in HISTORY_BY_KIND:
        raise PlanError(f"{name}: dataset kind {kind!r} is not one of {tuple(HISTORY_BY_KIND)} (ADR-016)")
    history = str(hist.get("type") or HISTORY_BY_KIND[kind])
    if history not in HISTORY_TYPES:
        raise PlanError(f"{name}: history type {history!r} is not one of {HISTORY_TYPES}")
    load, dedup = m.get("load") or {}, m.get("dedup") or {}
    order_by = str(load.get("history_order_by") or dedup.get("order_by") or "")
    if history in ("scd1", "scd2") and not order_by:
        raise PlanError(f"{name}: {history} needs an order (load.history_order_by or dedup.order_by)")
    names = {c.name for c in columns}
    track = tuple(hist.get("track_columns") or ())
    missing = sorted((set(track) | ({order_by} - {""})) - names)
    if missing:
        raise PlanError(f"{name}: history column(s) {', '.join(missing)} are not contract columns")
    valid_from, valid_to, gaps = _validity(name, history, hist, view.get("settings") or {}, names)
    deletes = load.get("deletes") or {}
    if str(deletes.get("apply", "none")) not in DELETE_APPLY:
        raise PlanError(f"{name}: deletes.apply {deletes.get('apply')!r} is not one of {DELETE_APPLY}")
    mode = str(dedup.get("mode", "none"))
    if mode not in DEDUP_MODES:
        raise PlanError(f"{name}: dedup.mode {mode!r} is not one of {DEDUP_MODES}")
    settings = view.get("settings") or {}
    disabled = frozenset(s for s, group in SWITCHES.items() if settings.get(f"processor_groups.{group}") == "disabled")
    if projection not in ("all", "declared"):
  1. Stray column settings. The manifest has settings for a column the contract does not have. Refused.
  2. No primary key. The contract names no primary key. Silver cannot identify a row without one. Refused.
  3. Unknown kind or history type. The kind is not one of the four, or the history type is not one of the five. Refused.
  4. No order for keyed history. Type 1 and type 2 need an order column, because changes must be laid out in the source's order, not in arrival order. Refused if neither order setting is present.
  5. History columns not in the contract. A tracked column or the order column is not a contract column. Refused.
  6. Unknown delete or duplicate setting. The value is not in the allowed list. Refused.
  7. Stage switches. For each of the six optional stages, the plan reads its processor_groups setting. The word disabled switches the stage off.
src/dataplatform/silver/plan.py:48The six stages a dataset may switch off, and the setting that does it. The other seven always run.
SWITCHES = {
    "hygiene": "hygiene",
    "parse_type": "type_safety",
    "time": "timestamp_normalize",
    "standardise": "standardization",
    "personal_data": "pii",
    "duplicates": "dedup",
}

Two times the plan works out for you

The plan decides two time columns so that the stages do not have to guess. The effective time says when a change took effect, and gives versions their validity. For a change feed it is the commit time the feed carried. Otherwise it is the order column when that column is a timestamp. The event time is the time of an appended event. It is the watermark column when that is a timestamp, and otherwise the effective time.

How it is tested

tests/dataplatform/test_silver_plan.py is one table of cases. It builds the plan for every tyre dataset, checks each refusal with its message, checks the stage order, and checks that the silver kernel imports no engine.

Not built yet, or known gaps
  • The dataset view does not carry the domain's quality defaults yet, so the plan falls back to defaults written beside it in the code.
  • Keeping only the contract's columns (projection declared) exists in the plan but has no setting in the manifest yet.