Data PlatformZoom level 3
Stage 7: personal data, tokens only
Some columns hold personal data, such as a dealer's contact address or an inspector's id. The contract marks them restricted. Ingestion replaces their values with tokens before bronze. A token is a stand-in text that is the same for the same value but cannot be turned back into it. The seventh stage is the guard behind that promise: it checks that every restricted column really holds tokens.
Without the guard, one misconfigured source would put plain personal data into silver, and from there into every table and report built on it, and nobody would notice.
A worked example
Three dealer rows. The dealer contract marks contact_email and contact_phone as restricted. The phone is not required, so it may be missing.
| dealer_code | contact_email | contact_phone | Result |
|---|---|---|---|
| D-1001 | tok_5c1e09b7a2d44f6e8a013c7d | tok_9a40e1f2c3b4a5968778695a | Goes on. |
| D-1002 | tok_0d3b6f18c9e24a7fb1a2c3d4 | missing | Goes on. A missing value is not a plain value. |
| D-1003 | a plain e-mail address | tok_77aa01bc23de45f678901234 | Quarantined with the reason contact_email is not tokenised. |
The settings that drive it
| Setting | What it means | Allowed values | Default |
|---|---|---|---|
The contract's classification | Marks a column as holding personal data. | The stage acts on the value restricted only. | Empty. The column is not checked. |
| The token pattern | A regular expression that a token matches. It is not a dataset setting. The deployment's binding passes it to the silver run. | Any regular expression | Empty. With restricted columns and no pattern the stage stops the run. |
processor_groups.pii | A dataset setting in the metadata store. The value disabled makes the stage runner skip this stage for the dataset. | The settings registry allows enabled, disabled, opt_in and dataset_decides. The plan acts only on disabled. | Not set, so the stage runs |
data-platform-catalog/slices/02-tyre/contracts/dealer.odcs.yaml:21The two restricted columns in the dealer contract. - {name: "contact_email", logicalType: "string", required: true, classification: "restricted", description: "Personal data class email; tokenised before bronze."}
- {name: "contact_phone", logicalType: "string", required: false, classification: "restricted", description: "Personal data class phone; tokenised before bronze."}src/dataplatform/adapters/keyed_tokens.py:17The token format of the keyed tokenizer. PATTERN = r"^tok_[0-9a-f]{24}$" # what a token looks like: silver checks restricted columns hold only thesesrc/dataplatform/adapters/keyed_tokens.py:24How that tokenizer makes a token: the first 24 hex digits of a keyed hash, after a fixed prefix. def token(self, column_class: str, value: str) -> str:
digest = hmac.new(self._key, f"{column_class}\x1f{value}".encode(), hashlib.sha256).hexdigest()
return f"tok_{digest[:24]}"Read the pattern piece by piece. ^ is the start of the value. tok_ is the literal prefix. [0-9a-f]{24} is exactly twenty-four characters, each a digit or a letter from a to f. $ is the end. So a value is a token only if it is the prefix followed by exactly twenty-four hex digits and nothing else.
The code
src/dataplatform/stages/clean.py:187The whole stage.def personal_data(df: DataFrame, ctx: StageCtx) -> Out:
"""A restricted column must hold tokens, never the plain value (DPC-0070): a row with a plain value is quarantined.
The token format comes from the binding's tokenizer."""
restricted = [c for c in ctx.plan.kept if c.restricted]
if not restricted:
return Out(df)
if not ctx.plan.token_pattern:
raise StageError(f"{ctx.plan.table}: restricted column(s) and no token format bound to check them")
checks = [
(F.col(c.name).isNotNull() & ~F.col(c.name).rlike(ctx.plan.token_pattern), f"{c.name} is not tokenised")
for c in restricted
]
good, bad = quarantine_rows(df, reasons(checks), "personal_data")
return Out(good, bad)- Find the restricted columns. The kept columns whose classification is
restricted. If there are none, the frame is returned as it came. - Refuse to guess. If there are restricted columns but no token pattern, the stage raises a stage error. Passing the rows unchecked would be the silent failure the stage exists to prevent.
- Build one check per column. The check is true when the value is present and does not match the pattern.
rlikeis a match against a regular expression. The sign~means not. - Split.
quarantine_rowssets aside every row with at least one failed check. A row with two plain values gets both reasons.
The privacy scan beside the stage
The same folder holds a second, smaller function. It is not one of the thirteen stages. It is a scan that can be run over bronze and silver at any time and counts, for each restricted column, the values that are present and do not look like a token.
src/dataplatform/stages/privacy.py:13The scan: one pass over the frame, one count per column.def clear_counts(df: DataFrame, columns: list[str], pattern: str) -> dict[str, int]:
"""Values present in each column that do not match the token pattern."""
cols = [c for c in columns if c in df.columns]
if not cols:
return {}
row = df.agg(*[F.sum(F.when(F.col(c).isNotNull() & ~F.col(c).cast("string").rlike(pattern), 1).otherwise(0))
.alias(c) for c in cols]).first()
return {c: int((row[c] if row else 0) or 0) for c in cols}For each column it adds up 1 for a value that is present and does not match, and 0 otherwise. All columns are counted in a single aggregation, so the frame is read once. A column the frame does not have is left out of the answer. The stage stops bad rows on the way in. The scan proves afterwards that none is there.
How it is tested
tests/dataplatform/test_silver_stages.py has three cases for the stage: a token passes, a plain value is quarantined with its reason, and a missing value passes. A fourth test checks the refusal when no token format is bound. tests/dataplatform/test_privacy_stage.py has three cases for the scan, including a token of the wrong length.
tests/dataplatform/test_silver_stages.py:205The test body for the three cases.def test_personal_data(spark, case, value, held):
p = plan(columns=[("id", "string"), ("email", "string")], restricted=("email",))
df = run("project", frame(spark, "id string, email string", [("a", value)]), p).df
out = run("personal_data", df, p)
assert reasons(out) == held, case
assert out.df.count() == (0 if held else 1)tests/dataplatform/test_silver_stages.py:213Restricted columns and no pattern: the stage refuses.def test_personal_data_needs_a_token_format(spark):
p = plan_from_view(view(columns=[("id", "string"), ("email", "string")], restricted=("email",)), "")
with pytest.raises(StageError, match="no token format"):
run("personal_data", frame(spark, "id string, email string", [("a", "x")]), p)tests/dataplatform/test_privacy_stage.py:10The pattern and the sample token the scan's cases use.PATTERN = r"^tok_[0-9a-f]{24}$"
TOK = "tok_" + "a" * 24- The quarantine record keeps every column of the row, so the plain value that caused the quarantine is stored in the quarantine table as it arrived. The quarantine table therefore needs the same protection as the source. I found no code that masks it.
- The stage checks the shape of a value, not where it came from. A text that happens to look like a token passes.
- Only the classification
restrictedis acted on. - The stage does not create tokens. If ingestion did not tokenise, every row of the batch is quarantined.
- Switching the stage off with
processor_groups.piiremoves the check entirely. Nothing else in silver looks at restricted columns during a run.