cleandataengineering.aiData Platform, from the big picture to the code
You are here: Start here › Bronze to silver › The thirteen stages › 4 Parse and type

Data PlatformZoom level 3

Stage 4: parse and type, casting without losing the original

The fourth stage gives every contract column the type its contract states. Text that holds a number becomes a number, text that holds a date becomes a date.

The danger in casting is silence. A plain cast turns a value it cannot read into a missing value and says nothing. A quantity typed as 4O with the letter O would become an empty quantity, and the order's total would be wrong with no trace. This stage makes every such value visible: the row goes to quarantine with the original text, or the value is emptied, flagged and counted, as the column's setting says.

Rows, some columns as textafter hygiene4 Parse and typecast beside, compare, replaceTyped rowsvalues convertedQuarantinerows with a value thatdid not convertEmptied valueskept, flagged, counted
Most rows take the top path. The two dashed boxes are the two ways a failed cast is made visible.
  1. Skip what is already right. A column that already has the target type and no declared pattern is not touched.
  2. Cast beside the original. For every other column a new working column _cast_<name> holds the converted value. The original column is still there.
  3. Detect a failure. A cast failed when the original has a value and the converted column has none.
  4. Split. Rows with a failure in a column that says quarantine are set aside. Their record holds the original values.
  5. Flag the emptied. Rows with a failure in a column that says empty stay, and get a flag.
  6. Replace. In the rows that stay, each original column is replaced by its converted column.

A worked example

Four sales order lines arrive with the quantity and the promised date as text. Suppose promised_date has the pattern dd.MM.yyyy. The first table is what goes in.

order_idline_noordered_qtypromised_date
SO-100110'40''20.11.2026'
SO-100210'4.5''21.11.2026'
SO-100310'24.0''31.02.2026'
SO-100410missing'22.11.2026'

Two rows go on, now typed.

order_idline_noordered_qtypromised_date
SO-10011040 (whole number)2026-11-20 (date)
SO-100410missing2026-11-22 (date)

Two rows go to quarantine. The record keeps the text exactly as it arrived.

_stage_reason_record (shortened)
parse_typeordered_qty is not a integer{"order_id":"SO-1002", "ordered_qty":"4.5", ...}
parse_typepromised_date is not a date{"order_id":"SO-1003", "promised_date":"31.02.2026", ...}

Notice four things. A quantity of 4.5 is not rounded to 4. The text 24.0 would have converted to 24, but its row left because of the date. The thirty-first of February is not a date. A missing quantity is not a failed cast, because there was nothing to convert. Whether a missing quantity is acceptable is a question for the quality stage. The wording is not a integer is the code's own text.

The settings that drive it

One honest remark about the tyre example. None of the nine tyre dataset manifests has a columns list today, so every tyre column runs with the defaults. The settings below are read by the code and covered by the tests, and the example manifest lines on this page are typed for this page, not copied from the tyre manifests.

SettingWhat it meansAllowed valuesDefault
The contract's logicalTypeThe type the column must have in silver.string, integer, number, boolean, timestamp, dateNone. The contract must state it.
columns[].formatThe pattern of a date or timestamp sent as text.A date pattern such as dd.MM.yyyyEmpty. The engine's standard form is expected.
columns[].on_cast_failureWhat happens to a value that does not convert.quarantine: the row is set aside. empty: the value becomes missing, the row is flagged and counted.quarantine
processor_groups.type_safetyA dataset setting in the metadata store. The value disabled makes the stage runner skip this stage for the dataset.The settings registry allows enabled, disabled, opt_in and dataset_decides. The plan acts only on disabled.Not set, so the stage runs
src/dataplatform/silver/plan.py:188The plan refuses to empty a required column or a key column.
    failure = str(override.get("on_cast_failure", "quarantine"))
    if failure not in CAST_FAILURE:
        raise PlanError(f"column {c['name']}: on_cast_failure {failure!r} is not one of {CAST_FAILURE}")
    if failure == "empty" and (c.get("required") or c.get("primary_key_position")):
        raise PlanError(f"column {c['name']}: a required or key column cannot be emptied on a failed cast")

The second refusal protects the data. Emptying a required column would only move the failure to the quality stage, and emptying a key part would make the row impossible to identify.

The code: the target types

src/dataplatform/stages/clean.py:25The six contract types and the engine type each one becomes.
SPARK_TYPES: dict[str, DataType] = {
    "string": StringType(),
    "integer": LongType(),
    "number": DoubleType(),
    "boolean": BooleanType(),
    "timestamp": TimestampType(),
    "date": DateType(),
}

An integer in the contract is a 64-bit whole number in silver. A number is a double, which is a floating point number. There is no decimal type in this list.

The code: one value

src/dataplatform/stages/clean.py:106How one column is converted. Every branch gives null for a value it cannot read.
def cast_to(name: str, logical: str, fmt: str = "") -> Column:
    """The contract type, or null when the value does not convert. A fraction never becomes a whole number.
    try_cast and try_to_timestamp: a value that does not convert is null, never an error, whatever the engine."""
    col = F.col(name)
    if logical in ("date", "timestamp") and fmt:
        parsed = F.try_to_timestamp(col.cast("string"), F.lit(fmt))
        return parsed.cast("date") if logical == "date" else parsed
    if logical == "integer":
        d = _try_cast(name, "double")
        return F.when(d == F.floor(d), d.cast("long"))
    if logical == "boolean":
        t = F.lower(col.cast("string"))
        return F.when(t.isin("true", "1"), F.lit(True)).when(t.isin("false", "0"), F.lit(False))
    if logical == "string":
        return col.cast("string")
    return _try_cast(name, SQL_TYPES[logical])
src/dataplatform/stages/clean.py:94The cast that gives null instead of an error, on every engine, and the SQL name of each type.
def _try_cast(name: str, sql_type: str) -> Column:
    """CAST that gives null when the value does not convert, on every engine (Spark with ANSI on or off, DuckDB):
    try_cast. The column name is checked and quoted; the type is one of SQL_TYPES, never data."""
    if not _NAME.match(name):
        raise StageError(f"{name!r} is not a safe column name")
    return F.expr(f"try_cast(`{name}` AS {sql_type})")


SQL_TYPES = {"string": "string", "integer": "bigint", "number": "double", "boolean": "boolean",
             "timestamp": "timestamp", "date": "date"}
Contract typeHow the value is convertedWhat that means
date or timestamp with a patternThe value is read as text with try_to_timestamp and the pattern. For a date the result is then cut to the day.A text that does not fit the pattern, or names a day that does not exist, gives null.
integerThe value is first cast to a double. It is kept only when the double equals its own floor, which is the number without its fraction. Then it is cast to a whole number.'42' and '42.0' become 42. '4.5' gives null and is a failure. A plain cast to a whole number would have cut it to 4.
booleanThe value is made lower-case text. true and 1 become true. false and 0 become false. There is no other branch.Anything else, such as yes, gives null and is a failure.
stringA plain cast to text.This cannot fail.
number, and date or timestamp without a patterntry_cast to the engine type.The engine's own reading of the text. A value it cannot read gives null.

_try_cast builds a small SQL expression, because try_cast is the one form that returns null for a bad value on Spark with strict mode on, on Spark with it off, and on DuckDB. The column name goes into that expression, so it is checked against the safe-name pattern first and wrapped in backticks. The type comes from a fixed dictionary, never from data.

The code: the stage

src/dataplatform/stages/clean.py:124The stage. The original column stays until the split is done.
def parse_type(df: DataFrame, ctx: StageCtx) -> Out:
    """Cast each column to its contract type. A value that does not convert is quarantined, or emptied, flagged and
    counted when its column says on_cast_failure: empty (DQ-02): never silently emptied."""
    types = {f.name: f.dataType for f in df.schema.fields}
    work = df
    checks: list[tuple[Column, str]] = []
    emptied: list[tuple[Column, str]] = []
    for c in ctx.plan.kept:
        target = SPARK_TYPES[c.logical_type]
        if types.get(c.name) == target and not c.fmt:
            continue
        work = work.withColumn(f"_cast_{c.name}", cast_to(c.name, c.logical_type, c.fmt))
        failed = F.col(c.name).isNotNull() & F.col(f"_cast_{c.name}").isNull()
        if c.on_cast_failure == "empty":
            emptied.append((failed, f"emptied:{c.name}"))
        else:
            checks.append((failed, f"{c.name} is not a {c.logical_type}"))
    good, bad = quarantine_rows(work, reasons(checks), "parse_type", df.columns)
    good = add_flags(good, emptied)
    tally = {"emptied": good.where(F.size(F.filter("_dq_flags", lambda f: f.startswith("emptied:"))) > 0)}
    for c in ctx.plan.kept:
        if f"_cast_{c.name}" in work.columns:
            good = good.withColumn(c.name, F.col(f"_cast_{c.name}")).drop(f"_cast_{c.name}")
    return Out(good, bad, tally if emptied else {})
  1. Read the actual types. types maps each column of the frame to the type it has now.
  2. Decide per column. For each kept column the target type is looked up. If the column already has it and no pattern is declared, the loop moves on.
  3. Add the converted column. work gets a new column _cast_<name>. The original column is not overwritten. This is the line that makes everything else possible.
  4. Define failed. failed is true when the original is not null and the converted value is null. A null original is not a failure.
  5. Sort the failure into one of two lists. If the column says empty, the pair goes to emptied with the flag name emptied:<name>. Otherwise it goes to checks with the reason text.
  6. Split. quarantine_rows is called with df.columns as the record columns. df is the frame as it came in, so the record holds the original values and none of the working columns. This is how a failed cast keeps the value the source sent.
  7. Flag. add_flags appends emptied:<name> to _dq_flags on the good rows where that cast failed.
  8. Count. The tally emptied is the good rows that carry at least one flag starting with emptied:.
  9. Replace and tidy. For each column that was cast, the original is overwritten with the converted column and the working column is dropped. For an emptied value the converted column is null, so the value in silver is missing.
src/dataplatform/stages/common.py:96How a flag name is appended to a row's list of flags.
def add_flags(df: DataFrame, flags: list[tuple[Column, str]]) -> DataFrame:
    """Append flag names to _dq_flags where their condition holds."""
    if not flags:
        return df
    new = F.array_compact(F.array(*[F.when(cond, F.lit(name)) for cond, name in flags]))
    return df.withColumn("_dq_flags", F.concat(F.coalesce(F.col("_dq_flags"), no_flags()), new))

add_flags builds a list with one entry per flag: the name where the condition holds and null where it does not. array_compact removes the nulls. The result is joined to the flags the row already has.

How it is tested

tests/dataplatform/test_silver_stages.py has nine cases. Each gives a contract type, one raw text, the column's settings, the expected value, the expected quarantine reasons and the expected count of emptied values.

tests/dataplatform/test_silver_stages.py:126The nine cases. The second is the fraction that is never cut. The eighth is the emptied value.
        ("text to whole number", "integer", "42", {}, 42, [], 0),
        ("a fraction is never truncated", "integer", "4.5", {}, None, ["x is not a integer"], 0),
        ("text to number", "number", "4.5", {}, 4.5, [], 0),
        ("text to date", "date", "2026-11-09", {}, date(2026, 11, 9), [], 0),
        ("a date with a declared pattern", "date", "09.11.2026", {"format": "dd.MM.yyyy"}, date(2026, 11, 9), [], 0),
        ("a bad date is quarantined (DQ-02)", "date", "31.02.2026", {"format": "dd.MM.yyyy"}, None,
         ["x is not a date"], 0),
        ("text to boolean", "boolean", "true", {}, True, [], 0),
        ("an unconvertible value emptied when the column says so, and counted", "integer", "n/a",
         {"on_cast_failure": "empty"}, None, [], 1),
        ("null stays null, no failure", "integer", None, {}, None, [], 0),
tests/dataplatform/test_silver_stages.py:139The test body. An emptied value must also carry its flag.
def test_parse_type(spark, case, logical, raw, settings, want, quarantined, emptied):
    p = plan(columns=[("id", "string"), ("x", logical)], column_settings=[{"name": "x", **settings}])
    df = run("project", frame(spark, "id string, x string", [("a", raw)]), p).df
    out = run("parse_type", df, p)
    assert reasons(out) == quarantined, case
    assert rows(out.df, "x") == ([] if quarantined else [(want,)])
    assert (out.tally["emptied"].count() if "emptied" in out.tally else 0) == emptied
    if emptied:
        assert rows(out.df, "_dq_flags") == [(["emptied:x"],)]
Not built yet, or known gaps
  • A whole number goes through a double on its way. A double holds whole numbers exactly only up to about nine thousand million million (two to the power 53). By my reading of the code, a larger whole number sent as text could change in its last digits. There is no test case for it. A column that already arrives as a whole number is not affected, because it is skipped.
  • There is no decimal type. Money is a double in silver.
  • A boolean accepts only true, false, 1 and 0.
  • When the stage is switched off, nothing checks that bronze types match the contract.
  • Only contract columns are cast. Other source columns keep the type bronze gave them.