Data PlatformZoom level 3
Stage 2: project, choosing and naming the columns
The second stage decides which columns go on and what they are called. After it, every contract column has its contract name, whatever the source called it, and every later stage can work with contract names only.
By default no column is thrown away. A column the source added that the contract does not know still reaches silver. Without that rule a new source column would be lost until someone changed the contract and reloaded history.
select builds the new column list in a fixed order._dq_flags.A worked example
Take a dealer row. Suppose the source calls the dealer's name name, sends a new column segment that the contract does not have, and the contract's contact_phone is not wanted in silver. The settings for that would look like this.
ExamplePer-column settings in a dataset manifest. Typed for this page.columns:
- {name: dealer_name, from: name}
- {name: contact_phone, exclude: true}The first table is the bronze row. The second is the row after the stage.
| dealer_code | name | region | contact_email | contact_phone | updated_at | segment | _batch_key |
|---|---|---|---|---|---|---|---|
| D-1001 | Sharma Tyres | north | tok_5c1e... | tok_9a40... | 2026-11-09 06:00:00 | retail | b-17 |
| dealer_code | dealer_name | region | contact_email | updated_at | segment | _batch_key | _dq_flags |
|---|---|---|---|---|---|---|---|
| D-1001 | Sharma Tyres | north | tok_5c1e... | 2026-11-09 06:00:00 | retail | b-17 | [] |
Three things changed. name became dealer_name. contact_phone is gone. A new column _dq_flags holds an empty list. The column segment passed through although the contract does not list it.
The settings that drive it
One honest remark about the tyre example. None of the nine tyre dataset manifests has a columns list today, so every tyre column runs with the defaults. The settings below are read by the code and covered by the tests, and the example manifest lines on this page are typed for this page, not copied from the tyre manifests.
| Setting | What it means | Allowed values | Default |
|---|---|---|---|
columns[].from | The column's name in bronze. | A safe column name | The contract name |
columns[].path | Take the column out of the JSON text in _extra. _extra is where ingestion puts columns it was told to hold. | A JSON path | Empty |
columns[].exclude | Leave the column out of silver. | true or false | false |
| Projection mode | all keeps other source columns. declared keeps only the contract's columns. | all, declared | all |
The projection mode is an argument of plan_from_view. No manifest key sets it yet. This stage has no switch.
src/dataplatform/silver/plan.py:95Everything the plan knows about one column. Stages 1 to 7 read these fields.@dataclass(frozen=True)
class ColumnPlan:
name: str
logical_type: str
required: bool = False
classification: str = ""
source: str = "" # the bronze column it comes from when renamed ("" = same name)
path: str = "" # a JSON path into the held columns (_extra) to extract it from
exclude: bool = False
hygiene: bool = True
on_cast_failure: str = "quarantine"
fmt: str = "" # a date or timestamp pattern for text values
local_tz: str = "" # adds <name>_local, the time in this zone
case: str = "" # upper | lower
values: tuple[tuple[str, str], ...] = () # declared value mapThe code
src/dataplatform/stages/clean.py:54The whole stage. It ends in one select.def project(df: DataFrame, ctx: StageCtx) -> Out:
"""All columns by default (ADR-008): contract columns under their contract names (renamed or extracted from the
held columns), every other source column too unless the projection is 'declared'; excluded columns go."""
plan = ctx.plan
for c in plan.columns:
if not _NAME.match(c.name) or not _NAME.match(_source_name(c)):
raise StageError(f"{plan.table}: {c.name!r} is not a safe column name")
sources = {_source_name(c) for c in plan.columns} | {c.name for c in plan.columns}
picked = [
(F.get_json_object(F.col("_extra"), c.path) if c.path else F.col(_source_name(c))).alias(c.name)
for c in plan.kept
]
others = [F.col(x) for x in df.columns if x not in sources and not x.startswith("_")]
meta = [F.col(x) for x in df.columns if x.startswith("_") and x not in sources and x != "_dq_flags"]
flags = F.col("_dq_flags") if "_dq_flags" in df.columns else no_flags()
return Out(df.select(*picked, *(others if plan.projection == "all" else []), *meta, flags.alias("_dq_flags")))src/dataplatform/stages/clean.py:34The pattern a column name must match._NAME = re.compile(r"^[A-Za-z_][A-Za-z0-9_]{0,127}$")- Refuse unsafe names. Every contract name and every source name must start with a letter or an underscore and continue with letters, digits or underscores, at most 128 characters in all. Later stages put column names into SQL expressions, so a name with a space or a quote is refused here, with a stage error.
- Know which names are spoken for.
sourcesis the set of every contract column's source name and contract name, including the excluded columns. It is used below to decide what counts as another source column. - Pick the contract columns. For each kept column the stage takes the bronze column under its source name, or, when the column has a path, the value at that path inside
_extra. Either way it is given the contract name withalias. This is where the rename happens. - Collect the other source columns.
othersis every bronze column that is not spoken for and does not start with an underscore. An excluded column is spoken for, so it is neither picked nor counted among the others. That is how it disappears. - Carry the underscore columns.
metais every column that starts with an underscore, such as_batch_key,_source_fileand_extra. They always go on. - Start the flags. If the frame already has
_dq_flagsit is kept. Otherwise an empty list of text is created. Later stages append names to it. - Select. The final column list is the picked columns, then the others when the projection is
all, then the underscore columns, then_dq_flags.
A value extracted from _extra comes out as text, because get_json_object returns text. Stage 4 then casts it to the contract's type like any other column.
src/dataplatform/stages/common.py:90Why the empty list is written as SQL text.def no_flags() -> Column:
"""An empty list of flag names. Written as SQL text because an empty F.array() does not compile on every engine
the stages run on (docs/DUCKDB-SPIKE.md); Spark reads it the same way."""
return F.expr("CAST(array() AS array<string>)")How it is tested
tests/dataplatform/test_silver_stages.py has four cases in a table and two further tests. Each case lists the columns that must be in the result and the columns that must not.
tests/dataplatform/test_silver_stages.py:73The four cases: all columns, a rename, an exclusion, an extraction. ("all columns by default", "id string, v long, seq long, w string", [("a", 1, 1, "x")], {},
["id", "v", "seq", "w", "_batch_key", "_dq_flags"], []),
("rename from the source name", "id string, value long, seq long", [("a", 1, 1)],
{"column_settings": [{"name": "v", "from": "value"}]}, ["v"], ["value"]),
("exclude a column", "id string, v long, seq long", [("a", 1, 1)],
{"column_settings": [{"name": "seq", "exclude": True}]}, ["id", "v"], ["seq"]),
("extract from the held columns", "id string, seq long, _extra string", [("a", 1, '{"v": 7}')],
{"column_settings": [{"name": "v", "path": "$.v"}]}, ["v"], []),tests/dataplatform/test_silver_stages.py:88The declared mode drops the column the contract does not have.def test_project_declared_keeps_only_the_contract(spark):
p = plan_from_view(view(), TOKEN, projection="declared")
out = run("project", frame(spark, "id string, v long, seq long, w string", [("a", 1, 1, "x")]), p).df
assert "w" not in out.columns
assert "id" in out.columns- Keeping only the contract's columns exists in the code but no manifest key selects it. The design note
docs/W1C-SILVER.mdlists this as a gap in the manifest specification. - The manifest's
columnslist has no typed schema of its own. The plan refuses a key it does not know, which is the only check today. - A held column in
_extradoes not become a real silver column by itself. It needs a contract column with apath.