cleandataengineering.aiData Platform, from the big picture to the code
You are here: Start here › Bronze to silver › The thirteen stages › 1 Contract check

Data PlatformZoom level 3

Stage 1: contract check, are all the columns there

The first stage looks at the column names of the new bronze rows and compares them with the dataset's contract. The contract is the agreed list of columns, types and keys of the dataset. If a contract column is missing, the stage stops the dataset run and names the column.

Without this stage a missing column would surface later as an engine error in whichever stage first touched it, with a message about an unresolved column and no hint that the source changed.

New bronze rowscolumn names only1 Contract checkis every kept column here?The same rows, untouchedall columns presentStageErrorthe run fails, nothing is written
The stage never changes a row. It has only two outcomes.
Goes inThe new bronze rows and the plan's list of kept columns.
What happensThe frame's column names are compared with the names the plan expects. No row is read.
Comes outThe same frame, or an error that names the missing columns.

A worked example

The sales order line contract has thirteen columns. Suppose the bronze frame arrives without promised_date. The table shows three situations and what the stage does in each.

Bronze frame hasColumn settingsResult
All thirteen contract columns and an extra column sales_officeNonePasses. An extra column is not a fault. The next stage decides what happens to it.
Everything except promised_dateNoneStops with: erp.sales_order_line: contract column(s) promised_date missing from bronze (DPC-0008)
qty but no ordered_qtyordered_qty has from: qtyPasses. A renamed column is looked up by its source name.

The settings that drive it

One honest remark about the tyre example. None of the nine tyre dataset manifests has a columns list today, so every tyre column runs with the defaults. The settings below are read by the code and covered by the tests, and the example manifest lines on this page are typed for this page, not copied from the tyre manifests.

SettingWhat it meansAllowed valuesDefault
The contract's column listThe columns the stage expects.Set by the contract fileNone
columns[].fromThe column's name in bronze when it differs from the contract name. The stage looks for this name.A column nameThe contract name
columns[].pathThe column is extracted from the held columns in _extra. The stage then expects _extra, not the column.A JSON path such as $.reasonEmpty
columns[].excludeThe column is left out of silver, so the stage does not expect it.true or falsefalse

This stage has no switch. It always runs.

The code

def contract_check(df: DataFrame, ctx: StageCtx) -> Out:
    """Every contract column the plan keeps is in the batch (DPC-0008). A missing one stops the batch, naming it."""
    have = set(df.columns)
    missing = sorted(_source_name(c) for c in ctx.plan.kept if not c.path and _source_name(c) not in have)
    if ctx.plan.kept and any(c.path for c in ctx.plan.kept) and "_extra" not in have:
        missing.append("_extra")
    if missing:
        raise StageError(f"{ctx.plan.table}: contract column(s) {', '.join(missing)} missing from bronze (DPC-0008)")
    return Out(df)
  1. The names bronze has. have is the set of column names of the frame.
  2. The names that are missing. ctx.plan.kept is the contract's columns without the excluded ones. For each kept column that is not extracted by a path, the stage takes its source name and checks whether bronze has it. The helper _source_name returns the from name if one is set and the contract name otherwise.
  3. The held columns. If any kept column is extracted by a path, the frame must have the _extra column to extract from. If it does not, _extra is added to the missing list.
  4. Stop or pass. If anything is missing, the stage raises StageError with the table name and the sorted list. Otherwise it returns the frame as it came.
src/dataplatform/stages/clean.py:37The name a contract column has in bronze.
def _source_name(c: ColumnPlan) -> str:
    return c.source or c.name

The error ends the dataset run. The silver runner records it at the step silver.stages, marks the dataset run failed, writes nothing and leaves the bookmark where it was. The hub page shows that part of the runner.

How it is tested

tests/dataplatform/test_silver_stages.py has five cases for this stage. Each case gives the frame's columns, one row, the column settings and the expected error, or none.

tests/dataplatform/test_silver_stages.py:50The five cases. The third expects an error that names column v.
        ("every column present", "id string, v long, seq long", [("a", 1, 1)], {}, None),
        ("an extra column is fine (all columns, ADR-008)", "id string, v long, seq long, w string",
         [("a", 1, 1, "x")], {}, None),
        ("a missing column stops the batch, named", "id string, seq long", [("a", 1)], {}, "contract column\\(s\\) v"),
        ("a renamed column is looked up by its source name", "id string, value long, seq long", [("a", 1, 1)],
         {"column_settings": [{"name": "v", "from": "value"}]}, None),
        ("an excluded column is not required", "id string, seq long", [("a", 1)],
         {"column_settings": [{"name": "v", "exclude": True}]}, None),
Not built yet, or known gaps
  • The stage checks names only. A column that is present with the wrong type passes here and is handled by stage 4.
  • The check is on the frame as a whole, not on each batch. Bronze is one table, so a batch that lacks a column which the table already has shows up as missing values, not as a missing column. Those missing values are caught by the required-column rules of the quality stage. This is my reading of the code. I did not verify what ingestion checks per batch before bronze.
  • The stage does not look at the contract's version.