cleandataengineering.aiData Platform, from the big picture to the code
You are here: Start here › Bronze to silver › The thirteen stages › 7 Personal data

Data PlatformZoom level 3

Stage 7: personal data, tokens only

Some columns hold personal data, such as a dealer's contact address or an inspector's id. The contract marks them restricted. Ingestion replaces their values with tokens before bronze. A token is a stand-in text that is the same for the same value but cannot be turned back into it. The seventh stage is the guard behind that promise: it checks that every restricted column really holds tokens.

Without the guard, one misconfigured source would put plain personal data into silver, and from there into every table and report built on it, and nobody would notice.

Rows with restricted columnscontact_email, contact_phone7 Personal datadoes each value look like a token?Rows that hold tokensor nothing in the columnQuarantinerows with a plain value
The stage does not create tokens. It only checks them.
Goes inRows after standardising, and the token format the deployment uses.
What happensEach restricted column's value is matched against the token pattern.
Comes outRows whose restricted values are tokens or missing go on. The others go to quarantine.

A worked example

Three dealer rows. The dealer contract marks contact_email and contact_phone as restricted. The phone is not required, so it may be missing.

dealer_codecontact_emailcontact_phoneResult
D-1001tok_5c1e09b7a2d44f6e8a013c7dtok_9a40e1f2c3b4a5968778695aGoes on.
D-1002tok_0d3b6f18c9e24a7fb1a2c3d4missingGoes on. A missing value is not a plain value.
D-1003a plain e-mail addresstok_77aa01bc23de45f678901234Quarantined with the reason contact_email is not tokenised.

The settings that drive it

SettingWhat it meansAllowed valuesDefault
The contract's classificationMarks a column as holding personal data.The stage acts on the value restricted only.Empty. The column is not checked.
The token patternA regular expression that a token matches. It is not a dataset setting. The deployment's binding passes it to the silver run.Any regular expressionEmpty. With restricted columns and no pattern the stage stops the run.
processor_groups.piiA dataset setting in the metadata store. The value disabled makes the stage runner skip this stage for the dataset.The settings registry allows enabled, disabled, opt_in and dataset_decides. The plan acts only on disabled.Not set, so the stage runs
data-platform-catalog/slices/02-tyre/contracts/dealer.odcs.yaml:21The two restricted columns in the dealer contract.
      - {name: "contact_email", logicalType: "string", required: true, classification: "restricted", description: "Personal data class email; tokenised before bronze."}
      - {name: "contact_phone", logicalType: "string", required: false, classification: "restricted", description: "Personal data class phone; tokenised before bronze."}
src/dataplatform/adapters/keyed_tokens.py:17The token format of the keyed tokenizer.
    PATTERN = r"^tok_[0-9a-f]{24}$"  # what a token looks like: silver checks restricted columns hold only these
src/dataplatform/adapters/keyed_tokens.py:24How that tokenizer makes a token: the first 24 hex digits of a keyed hash, after a fixed prefix.
    def token(self, column_class: str, value: str) -> str:
        digest = hmac.new(self._key, f"{column_class}\x1f{value}".encode(), hashlib.sha256).hexdigest()
        return f"tok_{digest[:24]}"

Read the pattern piece by piece. ^ is the start of the value. tok_ is the literal prefix. [0-9a-f]{24} is exactly twenty-four characters, each a digit or a letter from a to f. $ is the end. So a value is a token only if it is the prefix followed by exactly twenty-four hex digits and nothing else.

The code

def personal_data(df: DataFrame, ctx: StageCtx) -> Out:
    """A restricted column must hold tokens, never the plain value (DPC-0070): a row with a plain value is quarantined.
    The token format comes from the binding's tokenizer."""
    restricted = [c for c in ctx.plan.kept if c.restricted]
    if not restricted:
        return Out(df)
    if not ctx.plan.token_pattern:
        raise StageError(f"{ctx.plan.table}: restricted column(s) and no token format bound to check them")
    checks = [
        (F.col(c.name).isNotNull() & ~F.col(c.name).rlike(ctx.plan.token_pattern), f"{c.name} is not tokenised")
        for c in restricted
    ]
    good, bad = quarantine_rows(df, reasons(checks), "personal_data")
    return Out(good, bad)
  1. Find the restricted columns. The kept columns whose classification is restricted. If there are none, the frame is returned as it came.
  2. Refuse to guess. If there are restricted columns but no token pattern, the stage raises a stage error. Passing the rows unchecked would be the silent failure the stage exists to prevent.
  3. Build one check per column. The check is true when the value is present and does not match the pattern. rlike is a match against a regular expression. The sign ~ means not.
  4. Split. quarantine_rows sets aside every row with at least one failed check. A row with two plain values gets both reasons.

The privacy scan beside the stage

The same folder holds a second, smaller function. It is not one of the thirteen stages. It is a scan that can be run over bronze and silver at any time and counts, for each restricted column, the values that are present and do not look like a token.

src/dataplatform/stages/privacy.py:13The scan: one pass over the frame, one count per column.
def clear_counts(df: DataFrame, columns: list[str], pattern: str) -> dict[str, int]:
    """Values present in each column that do not match the token pattern."""
    cols = [c for c in columns if c in df.columns]
    if not cols:
        return {}
    row = df.agg(*[F.sum(F.when(F.col(c).isNotNull() & ~F.col(c).cast("string").rlike(pattern), 1).otherwise(0))
                   .alias(c) for c in cols]).first()
    return {c: int((row[c] if row else 0) or 0) for c in cols}

For each column it adds up 1 for a value that is present and does not match, and 0 otherwise. All columns are counted in a single aggregation, so the frame is read once. A column the frame does not have is left out of the answer. The stage stops bad rows on the way in. The scan proves afterwards that none is there.

How it is tested

tests/dataplatform/test_silver_stages.py has three cases for the stage: a token passes, a plain value is quarantined with its reason, and a missing value passes. A fourth test checks the refusal when no token format is bound. tests/dataplatform/test_privacy_stage.py has three cases for the scan, including a token of the wrong length.

tests/dataplatform/test_silver_stages.py:205The test body for the three cases.
def test_personal_data(spark, case, value, held):
    p = plan(columns=[("id", "string"), ("email", "string")], restricted=("email",))
    df = run("project", frame(spark, "id string, email string", [("a", value)]), p).df
    out = run("personal_data", df, p)
    assert reasons(out) == held, case
    assert out.df.count() == (0 if held else 1)
tests/dataplatform/test_silver_stages.py:213Restricted columns and no pattern: the stage refuses.
def test_personal_data_needs_a_token_format(spark):
    p = plan_from_view(view(columns=[("id", "string"), ("email", "string")], restricted=("email",)), "")
    with pytest.raises(StageError, match="no token format"):
        run("personal_data", frame(spark, "id string, email string", [("a", "x")]), p)
tests/dataplatform/test_privacy_stage.py:10The pattern and the sample token the scan's cases use.
PATTERN = r"^tok_[0-9a-f]{24}$"
TOK = "tok_" + "a" * 24
Not built yet, or known gaps
  • The quarantine record keeps every column of the row, so the plain value that caused the quarantine is stored in the quarantine table as it arrived. The quarantine table therefore needs the same protection as the source. I found no code that masks it.
  • The stage checks the shape of a value, not where it came from. A text that happens to look like a token passes.
  • Only the classification restricted is acted on.
  • The stage does not create tokens. If ingestion did not tokenise, every row of the batch is quarantined.
  • Switching the stage off with processor_groups.pii removes the check entirely. Nothing else in silver looks at restricted columns during a run.