Data PlatformZoom level 3
Stage 3: hygiene, cleaning up text
The third stage cleans every text column of the contract. It removes characters nobody can see, trims spaces, and turns text that is empty into a missing value.
Without it, 'D-1001' and 'D-1001 ' would be two different dealers, a number sent as ' 42' might fail to convert, and an empty text would count as a value in every rule that asks whether a column is filled.
A worked example
Four dealer names as a source might send them, and what silver gets. The marks [BOM], [CR], [LF] and [BEL] stand for single invisible characters. A dot stands for a space.
| dealer_name in bronze | dealer_name after hygiene | Which step acted |
|---|---|---|
[BOM]Sharma·Tyres | Sharma·Tyres | Step 1 removed the byte-order mark. |
··Coastal···Wheels· | Coastal·Wheels | Step 4 trimmed the ends. Step 5 made three spaces one. |
Hill[BEL]top[CR][LF]Depot | Hilltop[LF]Depot | Step 2 removed the bell character. Step 3 made the line ending a single line feed. |
··· | missing value | Step 4 left an empty text. The last line turned it into null. |
The settings that drive it
One honest remark about the tyre example. None of the nine tyre dataset manifests has a columns list today, so every tyre column runs with the defaults. The settings below are read by the code and covered by the tests, and the example manifest lines on this page are typed for this page, not copied from the tyre manifests.
| Setting | What it means | Allowed values | Default |
|---|---|---|---|
columns[].hygiene | Whether this column is cleaned. A column whose exact text matters can opt out. | true or false | true |
processor_groups.hygiene | A dataset setting in the metadata store. The value disabled makes the stage runner skip this stage for the dataset. | The settings registry allows enabled, disabled, opt_in and dataset_decides. The plan acts only on disabled. | Not set, so the stage runs |
ExampleA column that opts out. Typed for this page.columns:
- {name: defect_code, hygiene: false}The code
src/dataplatform/stages/clean.py:84The stage: which columns are cleaned.def hygiene(df: DataFrame, ctx: StageCtx) -> Out:
types = column_types(df)
out = df
for c in ctx.plan.kept:
if c.hygiene and types.get(c.name) == "string":
out = out.withColumn(c.name, clean_text(F.col(c.name)))
return Out(out)The stage reads each column's actual type from the frame. A column is cleaned when three things are true: it is a kept contract column, its hygiene setting is on, and it is text right now. The third condition matters. A contract column that will become a number in stage 4 but arrives as text is cleaned too, so its spaces are gone before the cast. A column that already arrives as a number is left alone.
src/dataplatform/stages/clean.py:73The cleaning itself: one expression made of five replacements.def clean_text(col: Column) -> Column:
"""LakeIngestionFramework's default-on hygiene processors, as one column expression: byte-order mark, control
characters, line endings, leading and trailing whitespace, runs of spaces, and empty text to null."""
x = F.regexp_replace(col, "^", "")
x = F.regexp_replace(x, _CONTROL, "")
x = F.regexp_replace(x, "\r\n?", "\n")
x = F.regexp_replace(x, "^\\s+|\\s+$", "")
x = F.regexp_replace(x, "[ \\t]{2,}", " ")
return F.when(x == "", F.lit(None)).otherwise(x)src/dataplatform/stages/clean.py:33The set of control characters that step 2 removes._CONTROL = "[\\x00-\\x08\\x0b\\x0c\\x0e-\\x1f\\x7f]"
Each line of clean_text is one regular expression replacement applied to the result of the line before. A regular expression is a pattern that describes which characters to find. The table takes them one at a time.
| Step | Pattern | What it finds | Replaced with | Why |
|---|---|---|---|---|
| 1 | ^ followed by the byte-order mark | The byte-order mark, but only as the very first character. ^ means the start of the value. The mark is the invisible character U+FEFF that some programs write at the start of a file. It is typed directly in the source, so the code box above shows nothing between ^ and the quote. | Nothing | A file's first value would otherwise start with an invisible character and match nothing. |
| 2 | [\x00-\x08\x0b\x0c\x0e-\x1f\x7f] | Any one control character. The square brackets list ranges of character codes. The ranges leave out three codes on purpose: \x09 is the tab, \x0a is the line feed and \x0d is the carriage return. | Nothing | Control characters such as the bell or the null character carry no meaning in business text and break displays and exports. |
| 3 | \r\n? | A carriage return, followed by a line feed if there is one. The ? makes the line feed optional. So it finds the Windows line ending and the old Mac line ending. | One line feed | The same text typed on two systems becomes equal. |
| 4 | ^\s+|\s+$ | White space at the start, or white space at the end. \s+ is one or more spaces, tabs or line breaks. $ means the end of the value. The bar means or. | Nothing | Spaces at the ends are almost never intended and make equal keys unequal. |
| 5 | [ \t]{2,} | Two or more spaces or tabs in a row. {2,} means at least two. Line feeds are not in the brackets, so a break between lines inside the text is kept. | One space | Double spaces inside names are typing noise. |
| Last | x == '' | A value that is now empty. | Null | An empty text and a missing value should not be two different things downstream. |
The order of the steps is part of the design. Control characters are removed before the line endings are unified, and that is safe because the carriage return and the line feed are not in the control set. The ends are trimmed before runs of spaces are collapsed. The test for empty comes last, so a value made only of spaces, tabs or control characters ends as null.
Two small consequences follow from the patterns. A single tab inside a text stays a tab, because step 5 needs at least two characters. A run of tabs becomes one space, not one tab.
The key stage later joins key parts with the character U+001F. That character is inside the control set of step 2, so a cleaned text can never contain it. The keys page uses this fact.
How it is tested
tests/dataplatform/test_silver_stages.py has nine cases. Each gives one raw value, the expected value and the settings. The test runs the whole chain of stages, not the hygiene function alone, so the last case also proves that the switch really skips the stage.
tests/dataplatform/test_silver_stages.py:105The nine cases. The last two are the opt-out and the switch. ("trimmed", " a b ", "a b", {}),
("runs of spaces collapsed", "a b", "a b", {}),
("empty becomes null", " ", None, {}),
("control characters removed", "a\x07b", "ab", {}),
("byte-order mark removed", "ab", "ab", {}),
("line endings made \\n", "a\r\nb", "a\nb", {}),
("null stays null", None, None, {}),
("a column may opt out", " a ", " a ", {"column_settings": [{"name": "s", "hygiene": False}]}),
("the stage switched off leaves values", " a ", " a ", {"settings": {"processor_groups.hygiene": "disabled"}}),tests/dataplatform/test_silver_stages.py:116The test body: one row through run_stages.def test_hygiene(spark, case, raw, want, kw):
p = plan(columns=[("id", "string"), ("s", "string")], **kw)
out, _, _ = run_stages(p, frame(spark, "id string, s string", [("a", raw)]), None, "r", "2026-11-09")
assert rows(out, "s") == [(want,)], case- Unicode normalisation is not applied. The design note
docs/W1C-SILVER.mdsays there is no built-in function for it on Spark. Two texts that look the same but are built from different code points stay different. - Only the six kinds of white space of the regular expression class are trimmed. A non-breaking space (U+00A0) at the end of a value is, by my reading of the pattern, not removed. There is no test case for it.
- The byte-order mark is removed only at the very start of a value.
- Only contract columns are cleaned. Other source columns that pass through under the all-columns rule keep their raw text.
- The stage does not count how many values it changed, so the run log cannot say how dirty a batch was.