Data PlatformZoom level 3
Stage 4: parse and type, casting without losing the original
The fourth stage gives every contract column the type its contract states. Text that holds a number becomes a number, text that holds a date becomes a date.
The danger in casting is silence. A plain cast turns a value it cannot read into a missing value and says nothing. A quantity typed as 4O with the letter O would become an empty quantity, and the order's total would be wrong with no trace. This stage makes every such value visible: the row goes to quarantine with the original text, or the value is emptied, flagged and counted, as the column's setting says.
- Skip what is already right. A column that already has the target type and no declared pattern is not touched.
- Cast beside the original. For every other column a new working column
_cast_<name>holds the converted value. The original column is still there. - Detect a failure. A cast failed when the original has a value and the converted column has none.
- Split. Rows with a failure in a column that says quarantine are set aside. Their record holds the original values.
- Flag the emptied. Rows with a failure in a column that says empty stay, and get a flag.
- Replace. In the rows that stay, each original column is replaced by its converted column.
A worked example
Four sales order lines arrive with the quantity and the promised date as text. Suppose promised_date has the pattern dd.MM.yyyy. The first table is what goes in.
| order_id | line_no | ordered_qty | promised_date |
|---|---|---|---|
| SO-1001 | 10 | '40' | '20.11.2026' |
| SO-1002 | 10 | '4.5' | '21.11.2026' |
| SO-1003 | 10 | '24.0' | '31.02.2026' |
| SO-1004 | 10 | missing | '22.11.2026' |
Two rows go on, now typed.
| order_id | line_no | ordered_qty | promised_date |
|---|---|---|---|
| SO-1001 | 10 | 40 (whole number) | 2026-11-20 (date) |
| SO-1004 | 10 | missing | 2026-11-22 (date) |
Two rows go to quarantine. The record keeps the text exactly as it arrived.
| _stage | _reason | _record (shortened) |
|---|---|---|
| parse_type | ordered_qty is not a integer | {"order_id":"SO-1002", "ordered_qty":"4.5", ...} |
| parse_type | promised_date is not a date | {"order_id":"SO-1003", "promised_date":"31.02.2026", ...} |
Notice four things. A quantity of 4.5 is not rounded to 4. The text 24.0 would have converted to 24, but its row left because of the date. The thirty-first of February is not a date. A missing quantity is not a failed cast, because there was nothing to convert. Whether a missing quantity is acceptable is a question for the quality stage. The wording is not a integer is the code's own text.
The settings that drive it
One honest remark about the tyre example. None of the nine tyre dataset manifests has a columns list today, so every tyre column runs with the defaults. The settings below are read by the code and covered by the tests, and the example manifest lines on this page are typed for this page, not copied from the tyre manifests.
| Setting | What it means | Allowed values | Default |
|---|---|---|---|
The contract's logicalType | The type the column must have in silver. | string, integer, number, boolean, timestamp, date | None. The contract must state it. |
columns[].format | The pattern of a date or timestamp sent as text. | A date pattern such as dd.MM.yyyy | Empty. The engine's standard form is expected. |
columns[].on_cast_failure | What happens to a value that does not convert. | quarantine: the row is set aside. empty: the value becomes missing, the row is flagged and counted. | quarantine |
processor_groups.type_safety | A dataset setting in the metadata store. The value disabled makes the stage runner skip this stage for the dataset. | The settings registry allows enabled, disabled, opt_in and dataset_decides. The plan acts only on disabled. | Not set, so the stage runs |
src/dataplatform/silver/plan.py:188The plan refuses to empty a required column or a key column. failure = str(override.get("on_cast_failure", "quarantine"))
if failure not in CAST_FAILURE:
raise PlanError(f"column {c['name']}: on_cast_failure {failure!r} is not one of {CAST_FAILURE}")
if failure == "empty" and (c.get("required") or c.get("primary_key_position")):
raise PlanError(f"column {c['name']}: a required or key column cannot be emptied on a failed cast")The second refusal protects the data. Emptying a required column would only move the failure to the quality stage, and emptying a key part would make the row impossible to identify.
The code: the target types
src/dataplatform/stages/clean.py:25The six contract types and the engine type each one becomes.SPARK_TYPES: dict[str, DataType] = {
"string": StringType(),
"integer": LongType(),
"number": DoubleType(),
"boolean": BooleanType(),
"timestamp": TimestampType(),
"date": DateType(),
}An integer in the contract is a 64-bit whole number in silver. A number is a double, which is a floating point number. There is no decimal type in this list.
The code: one value
src/dataplatform/stages/clean.py:106How one column is converted. Every branch gives null for a value it cannot read.def cast_to(name: str, logical: str, fmt: str = "") -> Column:
"""The contract type, or null when the value does not convert. A fraction never becomes a whole number.
try_cast and try_to_timestamp: a value that does not convert is null, never an error, whatever the engine."""
col = F.col(name)
if logical in ("date", "timestamp") and fmt:
parsed = F.try_to_timestamp(col.cast("string"), F.lit(fmt))
return parsed.cast("date") if logical == "date" else parsed
if logical == "integer":
d = _try_cast(name, "double")
return F.when(d == F.floor(d), d.cast("long"))
if logical == "boolean":
t = F.lower(col.cast("string"))
return F.when(t.isin("true", "1"), F.lit(True)).when(t.isin("false", "0"), F.lit(False))
if logical == "string":
return col.cast("string")
return _try_cast(name, SQL_TYPES[logical])src/dataplatform/stages/clean.py:94The cast that gives null instead of an error, on every engine, and the SQL name of each type.def _try_cast(name: str, sql_type: str) -> Column:
"""CAST that gives null when the value does not convert, on every engine (Spark with ANSI on or off, DuckDB):
try_cast. The column name is checked and quoted; the type is one of SQL_TYPES, never data."""
if not _NAME.match(name):
raise StageError(f"{name!r} is not a safe column name")
return F.expr(f"try_cast(`{name}` AS {sql_type})")
SQL_TYPES = {"string": "string", "integer": "bigint", "number": "double", "boolean": "boolean",
"timestamp": "timestamp", "date": "date"}| Contract type | How the value is converted | What that means |
|---|---|---|
| date or timestamp with a pattern | The value is read as text with try_to_timestamp and the pattern. For a date the result is then cut to the day. | A text that does not fit the pattern, or names a day that does not exist, gives null. |
| integer | The value is first cast to a double. It is kept only when the double equals its own floor, which is the number without its fraction. Then it is cast to a whole number. | '42' and '42.0' become 42. '4.5' gives null and is a failure. A plain cast to a whole number would have cut it to 4. |
| boolean | The value is made lower-case text. true and 1 become true. false and 0 become false. There is no other branch. | Anything else, such as yes, gives null and is a failure. |
| string | A plain cast to text. | This cannot fail. |
| number, and date or timestamp without a pattern | try_cast to the engine type. | The engine's own reading of the text. A value it cannot read gives null. |
_try_cast builds a small SQL expression, because try_cast is the one form that returns null for a bad value on Spark with strict mode on, on Spark with it off, and on DuckDB. The column name goes into that expression, so it is checked against the safe-name pattern first and wrapped in backticks. The type comes from a fixed dictionary, never from data.
The code: the stage
src/dataplatform/stages/clean.py:124The stage. The original column stays until the split is done.def parse_type(df: DataFrame, ctx: StageCtx) -> Out:
"""Cast each column to its contract type. A value that does not convert is quarantined, or emptied, flagged and
counted when its column says on_cast_failure: empty (DQ-02): never silently emptied."""
types = {f.name: f.dataType for f in df.schema.fields}
work = df
checks: list[tuple[Column, str]] = []
emptied: list[tuple[Column, str]] = []
for c in ctx.plan.kept:
target = SPARK_TYPES[c.logical_type]
if types.get(c.name) == target and not c.fmt:
continue
work = work.withColumn(f"_cast_{c.name}", cast_to(c.name, c.logical_type, c.fmt))
failed = F.col(c.name).isNotNull() & F.col(f"_cast_{c.name}").isNull()
if c.on_cast_failure == "empty":
emptied.append((failed, f"emptied:{c.name}"))
else:
checks.append((failed, f"{c.name} is not a {c.logical_type}"))
good, bad = quarantine_rows(work, reasons(checks), "parse_type", df.columns)
good = add_flags(good, emptied)
tally = {"emptied": good.where(F.size(F.filter("_dq_flags", lambda f: f.startswith("emptied:"))) > 0)}
for c in ctx.plan.kept:
if f"_cast_{c.name}" in work.columns:
good = good.withColumn(c.name, F.col(f"_cast_{c.name}")).drop(f"_cast_{c.name}")
return Out(good, bad, tally if emptied else {})- Read the actual types.
typesmaps each column of the frame to the type it has now. - Decide per column. For each kept column the target type is looked up. If the column already has it and no pattern is declared, the loop moves on.
- Add the converted column.
workgets a new column_cast_<name>. The original column is not overwritten. This is the line that makes everything else possible. - Define failed.
failedis true when the original is not null and the converted value is null. A null original is not a failure. - Sort the failure into one of two lists. If the column says
empty, the pair goes toemptiedwith the flag nameemptied:<name>. Otherwise it goes tocheckswith the reason text. - Split.
quarantine_rowsis called withdf.columnsas the record columns.dfis the frame as it came in, so the record holds the original values and none of the working columns. This is how a failed cast keeps the value the source sent. - Flag.
add_flagsappendsemptied:<name>to_dq_flagson the good rows where that cast failed. - Count. The tally
emptiedis the good rows that carry at least one flag starting withemptied:. - Replace and tidy. For each column that was cast, the original is overwritten with the converted column and the working column is dropped. For an emptied value the converted column is null, so the value in silver is missing.
src/dataplatform/stages/common.py:96How a flag name is appended to a row's list of flags.def add_flags(df: DataFrame, flags: list[tuple[Column, str]]) -> DataFrame:
"""Append flag names to _dq_flags where their condition holds."""
if not flags:
return df
new = F.array_compact(F.array(*[F.when(cond, F.lit(name)) for cond, name in flags]))
return df.withColumn("_dq_flags", F.concat(F.coalesce(F.col("_dq_flags"), no_flags()), new))add_flags builds a list with one entry per flag: the name where the condition holds and null where it does not. array_compact removes the nulls. The result is joined to the flags the row already has.
How it is tested
tests/dataplatform/test_silver_stages.py has nine cases. Each gives a contract type, one raw text, the column's settings, the expected value, the expected quarantine reasons and the expected count of emptied values.
tests/dataplatform/test_silver_stages.py:126The nine cases. The second is the fraction that is never cut. The eighth is the emptied value. ("text to whole number", "integer", "42", {}, 42, [], 0),
("a fraction is never truncated", "integer", "4.5", {}, None, ["x is not a integer"], 0),
("text to number", "number", "4.5", {}, 4.5, [], 0),
("text to date", "date", "2026-11-09", {}, date(2026, 11, 9), [], 0),
("a date with a declared pattern", "date", "09.11.2026", {"format": "dd.MM.yyyy"}, date(2026, 11, 9), [], 0),
("a bad date is quarantined (DQ-02)", "date", "31.02.2026", {"format": "dd.MM.yyyy"}, None,
["x is not a date"], 0),
("text to boolean", "boolean", "true", {}, True, [], 0),
("an unconvertible value emptied when the column says so, and counted", "integer", "n/a",
{"on_cast_failure": "empty"}, None, [], 1),
("null stays null, no failure", "integer", None, {}, None, [], 0),tests/dataplatform/test_silver_stages.py:139The test body. An emptied value must also carry its flag.def test_parse_type(spark, case, logical, raw, settings, want, quarantined, emptied):
p = plan(columns=[("id", "string"), ("x", logical)], column_settings=[{"name": "x", **settings}])
df = run("project", frame(spark, "id string, x string", [("a", raw)]), p).df
out = run("parse_type", df, p)
assert reasons(out) == quarantined, case
assert rows(out.df, "x") == ([] if quarantined else [(want,)])
assert (out.tally["emptied"].count() if "emptied" in out.tally else 0) == emptied
if emptied:
assert rows(out.df, "_dq_flags") == [(["emptied:x"],)]- A whole number goes through a double on its way. A double holds whole numbers exactly only up to about nine thousand million million (two to the power 53). By my reading of the code, a larger whole number sent as text could change in its last digits. There is no test case for it. A column that already arrives as a whole number is not affected, because it is skipped.
- There is no decimal type. Money is a double in silver.
- A boolean accepts only true, false, 1 and 0.
- When the stage is switched off, nothing checks that bronze types match the contract.
- Only contract columns are cast. Other source columns keep the type bronze gave them.