cleandataengineering.aiData Platform, from the big picture to the code
You are here: Start here › Bronze to silver › The thirteen stages › 2 Project

Data PlatformZoom level 3

Stage 2: project, choosing and naming the columns

The second stage decides which columns go on and what they are called. After it, every contract column has its contract name, whatever the source called it, and every later stage can work with contract names only.

By default no column is thrown away. A column the source added that the contract does not know still reaches silver. Without that rule a new source column would be lost until someone changed the contract and reloaded history.

Bronze rowssource names, load columns2 Projectselect, rename, extractRows with contract namesplus other source columns,load columns, _dq_flags
Rows are not added or removed. Only the columns change.
Goes inBronze rows with the source's column names and the columns the lander stamped.
What happensOne select builds the new column list in a fixed order.
Comes outThe same rows. Contract columns first, then other source columns, then the underscore columns, then _dq_flags.

A worked example

Take a dealer row. Suppose the source calls the dealer's name name, sends a new column segment that the contract does not have, and the contract's contact_phone is not wanted in silver. The settings for that would look like this.

ExamplePer-column settings in a dataset manifest. Typed for this page.
columns:
  - {name: dealer_name, from: name}
  - {name: contact_phone, exclude: true}

The first table is the bronze row. The second is the row after the stage.

dealer_codenameregioncontact_emailcontact_phoneupdated_atsegment_batch_key
D-1001Sharma Tyresnorthtok_5c1e...tok_9a40...2026-11-09 06:00:00retailb-17
dealer_codedealer_nameregioncontact_emailupdated_atsegment_batch_key_dq_flags
D-1001Sharma Tyresnorthtok_5c1e...2026-11-09 06:00:00retailb-17[]

Three things changed. name became dealer_name. contact_phone is gone. A new column _dq_flags holds an empty list. The column segment passed through although the contract does not list it.

The settings that drive it

One honest remark about the tyre example. None of the nine tyre dataset manifests has a columns list today, so every tyre column runs with the defaults. The settings below are read by the code and covered by the tests, and the example manifest lines on this page are typed for this page, not copied from the tyre manifests.

SettingWhat it meansAllowed valuesDefault
columns[].fromThe column's name in bronze.A safe column nameThe contract name
columns[].pathTake the column out of the JSON text in _extra. _extra is where ingestion puts columns it was told to hold.A JSON pathEmpty
columns[].excludeLeave the column out of silver.true or falsefalse
Projection modeall keeps other source columns. declared keeps only the contract's columns.all, declaredall

The projection mode is an argument of plan_from_view. No manifest key sets it yet. This stage has no switch.

src/dataplatform/silver/plan.py:95Everything the plan knows about one column. Stages 1 to 7 read these fields.
@dataclass(frozen=True)
class ColumnPlan:
    name: str
    logical_type: str
    required: bool = False
    classification: str = ""
    source: str = ""  # the bronze column it comes from when renamed ("" = same name)
    path: str = ""  # a JSON path into the held columns (_extra) to extract it from
    exclude: bool = False
    hygiene: bool = True
    on_cast_failure: str = "quarantine"
    fmt: str = ""  # a date or timestamp pattern for text values
    local_tz: str = ""  # adds <name>_local, the time in this zone
    case: str = ""  # upper | lower
    values: tuple[tuple[str, str], ...] = ()  # declared value map

The code

src/dataplatform/stages/clean.py:54The whole stage. It ends in one select.
def project(df: DataFrame, ctx: StageCtx) -> Out:
    """All columns by default (ADR-008): contract columns under their contract names (renamed or extracted from the
    held columns), every other source column too unless the projection is 'declared'; excluded columns go."""
    plan = ctx.plan
    for c in plan.columns:
        if not _NAME.match(c.name) or not _NAME.match(_source_name(c)):
            raise StageError(f"{plan.table}: {c.name!r} is not a safe column name")
    sources = {_source_name(c) for c in plan.columns} | {c.name for c in plan.columns}
    picked = [
        (F.get_json_object(F.col("_extra"), c.path) if c.path else F.col(_source_name(c))).alias(c.name)
        for c in plan.kept
    ]
    others = [F.col(x) for x in df.columns if x not in sources and not x.startswith("_")]
    meta = [F.col(x) for x in df.columns if x.startswith("_") and x not in sources and x != "_dq_flags"]
    flags = F.col("_dq_flags") if "_dq_flags" in df.columns else no_flags()
    return Out(df.select(*picked, *(others if plan.projection == "all" else []), *meta, flags.alias("_dq_flags")))
src/dataplatform/stages/clean.py:34The pattern a column name must match.
_NAME = re.compile(r"^[A-Za-z_][A-Za-z0-9_]{0,127}$")
  1. Refuse unsafe names. Every contract name and every source name must start with a letter or an underscore and continue with letters, digits or underscores, at most 128 characters in all. Later stages put column names into SQL expressions, so a name with a space or a quote is refused here, with a stage error.
  2. Know which names are spoken for. sources is the set of every contract column's source name and contract name, including the excluded columns. It is used below to decide what counts as another source column.
  3. Pick the contract columns. For each kept column the stage takes the bronze column under its source name, or, when the column has a path, the value at that path inside _extra. Either way it is given the contract name with alias. This is where the rename happens.
  4. Collect the other source columns. others is every bronze column that is not spoken for and does not start with an underscore. An excluded column is spoken for, so it is neither picked nor counted among the others. That is how it disappears.
  5. Carry the underscore columns. meta is every column that starts with an underscore, such as _batch_key, _source_file and _extra. They always go on.
  6. Start the flags. If the frame already has _dq_flags it is kept. Otherwise an empty list of text is created. Later stages append names to it.
  7. Select. The final column list is the picked columns, then the others when the projection is all, then the underscore columns, then _dq_flags.

A value extracted from _extra comes out as text, because get_json_object returns text. Stage 4 then casts it to the contract's type like any other column.

src/dataplatform/stages/common.py:90Why the empty list is written as SQL text.
def no_flags() -> Column:
    """An empty list of flag names. Written as SQL text because an empty F.array() does not compile on every engine
    the stages run on (docs/DUCKDB-SPIKE.md); Spark reads it the same way."""
    return F.expr("CAST(array() AS array<string>)")

How it is tested

tests/dataplatform/test_silver_stages.py has four cases in a table and two further tests. Each case lists the columns that must be in the result and the columns that must not.

tests/dataplatform/test_silver_stages.py:73The four cases: all columns, a rename, an exclusion, an extraction.
        ("all columns by default", "id string, v long, seq long, w string", [("a", 1, 1, "x")], {},
         ["id", "v", "seq", "w", "_batch_key", "_dq_flags"], []),
        ("rename from the source name", "id string, value long, seq long", [("a", 1, 1)],
         {"column_settings": [{"name": "v", "from": "value"}]}, ["v"], ["value"]),
        ("exclude a column", "id string, v long, seq long", [("a", 1, 1)],
         {"column_settings": [{"name": "seq", "exclude": True}]}, ["id", "v"], ["seq"]),
        ("extract from the held columns", "id string, seq long, _extra string", [("a", 1, '{"v": 7}')],
         {"column_settings": [{"name": "v", "path": "$.v"}]}, ["v"], []),
tests/dataplatform/test_silver_stages.py:88The declared mode drops the column the contract does not have.
def test_project_declared_keeps_only_the_contract(spark):
    p = plan_from_view(view(), TOKEN, projection="declared")
    out = run("project", frame(spark, "id string, v long, seq long, w string", [("a", 1, 1, "x")]), p).df
    assert "w" not in out.columns
    assert "id" in out.columns
Not built yet, or known gaps
  • Keeping only the contract's columns exists in the code but no manifest key selects it. The design note docs/W1C-SILVER.md lists this as a gap in the manifest specification.
  • The manifest's columns list has no typed schema of its own. The plan refuses a key it does not know, which is the only check today.
  • A held column in _extra does not become a real silver column by itself. It needs a contract column with a path.