cleandataengineering.aiData Platform, from the big picture to the code
You are here: Start here › Code map › Who calls what › stages

Data PlatformZoom level 3

Who calls what: stages

Every class, function and method in the stages package, with who calls it. Found by reading the code. A call written as object.method() is matched by the method's name, so a same-named method elsewhere can appear: those rows say by name.

54 definitions in 7 modules. Jump to a module: __init__.py · baseline.py · clean.py · common.py · history.py · privacy.py · rows.py

stages/__init__.py

The thirteen standard bronze-to-silver stages (ADR-007) as pure PySpark functions: DataFrames in, DataFrames out, no reads or writes (ADR-013). This is the in-house part of the stage_runner port the engine binding names (dbs/engine/databric The thirteen stages

DefinitionWhat it does (its own first sentence)Called byNamed in tests
run_stages
function, line 53
Every stage in order; a stage the plan switches off is skipped. Returns the rows to write, the quarantined rows of every stage, and the frames to count, named <stage>.<what>. `hold`, when the engine adapter passes one, fixes each stage's oudeclare.staged · realms/databricks/declarative/silver.py:42
DuckStageRunner.run · adapters/duckdb_silver.py:67
SparkStageRunner.run · adapters/spark_silver.py:103
test_baseline_stage.py, test_silver_stages.py

stages/baseline.py

The baseline batch (B8, ADR-029, assumed decision A3): a compaction of bronze, not a copy of silver. The thirteen stages

DefinitionWhat it does (its own first sentence)Called byNamed in tests
compactable
function, line 27
'' when the dataset can have a baseline, else the reason it keeps its bronze.SilverBaseliner.write · silver/baseline_writer.py:53
compact · stages/baseline.py:47
test_baseline_stage.py
_bronze_name
function, line 41
compact · stages/baseline.py:50no test names it
compact
function, line 45
The last row per business key (history order first, then arrival), in bronze's own columns.SilverBaseliner.write · silver/baseline_writer.py:63test_baseline_stage.py
_live
function, line 58
replay_check · stages/baseline.py:72no test names it
replay_check
function, line 68
Live keys of the replayed baseline against live keys of current silver, leaving out keys any later batch touched (their silver rows moved on after the cut-off). Every frame carries _key_hash (the keys processor). Returns baseline_keys, silvSilverBaseliner._check · silver/baseline_writer.py:86test_baseline_stage.py

stages/clean.py

Stages 1-9: contract check, projection, hygiene, parse and type, time, standardise, personal data, row filter, quality. Each is a pure function over a DataFrame (ADR-013); the settings come from the plan (ADR-007). The thirteen stages

DefinitionWhat it does (its own first sentence)Called byNamed in tests
_source_name
function, line 37
contract_check · stages/clean.py:45
project · stages/clean.py:59
no test names it
contract_check
function, line 42
Every contract column the plan keeps is in the batch (DPC-0008). A missing one stops the batch, naming it.module level · stages/__init__.py:34 (value)test_silver_stages.py
project
function, line 54
All columns by default (ADR-008): contract columns under their contract names (renamed or extracted from the held columns), every other source column too unless the projection is 'declared'; excluded columns go.module level · stages/__init__.py:35 (value)test_cloud_adapters.py, test_lakebase.py and 6 more
clean_text
function, line 73
LakeIngestionFramework's default-on hygiene processors, as one column expression: byte-order mark, control characters, line endings, leading and trailing whitespace, runs of spaces, and empty text to null.hygiene · stages/clean.py:89no test names it
hygiene
function, line 84
module level · stages/__init__.py:36 (value)test_silver_declarative.py, test_silver_plan.py and 6 more
_try_cast
function, line 94
CAST that gives null when the value does not convert, on every engine (Spark with ANSI on or off, DuckDB): try_cast. The column name is checked and quoted; the type is one of SQL_TYPES, never data.cast_to · stages/clean.py:114no test names it
cast_to
function, line 106
The contract type, or null when the value does not convert. A fraction never becomes a whole number. try_cast and try_to_timestamp: a value that does not convert is null, never an error, whatever the engine.parse_type · stages/clean.py:135no test names it
parse_type
function, line 124
Cast each column to its contract type. A value that does not convert is quarantined, or emptied, flagged and counted when its column says on_cast_failure: empty (DQ-02): never silently emptied.module level · stages/__init__.py:37 (value)test_silver_plan.py, test_silver_stages.py
time
function, line 151
Timestamps stay instants in UTC (the session's zone). A column with local_tz gets <name>_local beside it: the wall-clock time in that zone, as text (ISO, no offset).module level · stages/__init__.py:38 (value)no test names it
standardise
function, line 167
Declared value maps (a value not in the map is kept as it is) and letter case.module level · stages/__init__.py:39 (value)test_silver_stages.py
personal_data
function, line 187
A restricted column must hold tokens, never the plain value (DPC-0070): a row with a plain value is quarantined. The token format comes from the binding's tokenizer.module level · stages/__init__.py:40 (value)test_silver_stages.py
row_filter
function, line 204
Declared filters: rows that do not satisfy one are dropped and counted (not quarantined: they are not wrong).module level · stages/__init__.py:41 (value)test_silver_stages.py
quality
function, line 216
Row rules; severity sets the action (ADR-009): quarantine the row, or flag it in _dq_flags and keep it.module level · stages/__init__.py:42 (value)silver_helpers.py, test_access_model.py and 12 more

stages/common.py

What every stage shares: its inputs, its output, and the quarantine record shape. The thirteen stages

DefinitionWhat it does (its own first sentence)Called byNamed in tests
StageCtx
class, line 23
run_stages · stages/__init__.py:66
module level · stages/common.py:38 (value)
test_silver_stages.py
Out
class, line 32
contract_check · stages/clean.py:50
project · stages/clean.py:69
hygiene · stages/clean.py:90
parse_type · stages/clean.py:147
time · stages/clean.py:163
standardise · stages/clean.py:183
and 8 more
no test names it
quarantine_rows
function, line 41
Split a frame on `reason` (null = good row): (good rows, quarantine records keeping every original value of the `record` columns, all columns by default).parse_type · stages/clean.py:141
personal_data · stages/clean.py:199
quality · stages/clean.py:225
source_versioned · stages/history.py:291
keys · stages/rows.py:67
no test names it
reasons
function, line 59
One text naming every failed check, or null when none failed.parse_type · stages/clean.py:141
personal_data · stages/clean.py:199
quality · stages/clean.py:225
keys · stages/rows.py:67
test_contract_rules.py, test_hop_reports.py and 3 more
null_mark
function, line 68
The text standing for a null key part in a hash: one NUL character. Built by chr(0) rather than a literal so every engine reads the same character (a NUL inside a SQL string literal does not survive on all of them; docs/DUCKDB-SPIKE.md). Thtext_parts · stages/common.py:77test_silver_stages.py
text_parts
function, line 75
The columns as one text, SEP between parts, a null part as null_mark().key_hash · stages/common.py:81
scd2 · stages/history.py:145
test_silver_stages.py
key_hash
function, line 80
keys · stages/rows.py:68no test names it
column_types
function, line 84
Each column's type in Spark's simple form (string, bigint, timestamp, ...), read from the schema (portable: `DataFrame.dtypes` is not on every engine).hygiene · stages/clean.py:85test_obs_dashboards.py, test_ops_export.py
no_flags
function, line 90
An empty list of flag names. Written as SQL text because an empty F.array() does not compile on every engine the stages run on (docs/DUCKDB-SPIKE.md); Spark reads it the same way.project · stages/clean.py:68
add_flags · stages/common.py:101
no test names it
add_flags
function, line 96
Append flag names to _dq_flags where their condition holds.parse_type · stages/clean.py:142
quality · stages/clean.py:226
no test names it
arrival_order
function, line 107
How rows arrived, oldest first: feed position, stream sequence, arrival time, file name, batch.duplicates · stages/rows.py:38no test names it
arrival_desc
function, line 112
Newest arrival first: the latest delivery of the same change wins a tie.compact · stages/baseline.py:54
scd2 · stages/history.py:158
scd1 · stages/history.py:182
source_versioned · stages/history.py:273
duplicates · stages/rows.py:49
no test names it

stages/history.py

Stage 12: history by dataset kind (ADR-016), deletes soft. A pure function: new rows and silver as it stands in, the rows to write out. Every output row has a `_row_key` and an `_action`: `upsert` (write it) or `delete` (a stale row of a ke The thirteen stages

DefinitionWhat it does (its own first sentence)Called byNamed in tests
working
function, line 32
Silver rows in the stages' working shape (bronze run under its bronze name, silver-only columns gone).snapshot · stages/history.py:236
history · stages/history.py:314
test_gold_dbt_parse.py
_ts
function, line 40
_incoming · stages/history.py:51
append · stages/history.py:213
no test names it
_op
function, line 46
_incoming · stages/history.py:51
_existing · stages/history.py:59
no test names it
_incoming
function, line 50
scd2 · stages/history.py:139
scd1 · stages/history.py:180
no test names it
_existing
function, line 58
_key_compare_deletes · stages/history.py:87
_events · stages/history.py:109
no test names it
_key_compare_deletes
function, line 69
A full copy is the whole source: a key live in silver and missing from the copy was deleted at the source (DPC-0017). The delete takes the copy's order value and time. The mass-delete guard has already held any copy missing too many keys (D_events · stages/history.py:105no test names it
_events
function, line 101
(every event of the keys this batch touches, silver's rows of those keys).scd2 · stages/history.py:140
scd1 · stages/history.py:180
no test names it
_stale
function, line 112
Silver rows of the touched keys whose row key the new history no longer has: removed.scd2 · stages/history.py:174
scd1 · stages/history.py:193
source_versioned · stages/history.py:306
no test names it
_finish
function, line 120
scd2 · stages/history.py:174
scd1 · stages/history.py:193
append · stages/history.py:214
snapshot · stages/history.py:235
source_versioned · stages/history.py:307
no test names it
_tracked
function, line 130
scd2 · stages/history.py:145no test names it
scd2
function, line 137
module level · stages/history.py:310 (value)no test names it
scd1
function, line 178
module level · stages/history.py:310 (value)no test names it
_live
function, line 197
A row that is current, not deleted, with no version history of its own.append · stages/history.py:214
snapshot · stages/history.py:231
no test names it
append
function, line 209
module level · stages/history.py:310 (value)azure_fakes.py, gold_helpers.py and 45 more
snapshot
function, line 218
module level · stages/history.py:310 (value)test_meta_loader.py, test_obs_recorder.py and 10 more
source_versioned
function, line 255
History the source already keeps (ADR-032): every version as delivered, with the source's own validity dates, checked per key and never rebuilt. A key whose versions overlap, leave a gap (unless gaps are allowed) or end before they start ishistory · stages/history.py:316test_silver_declarative.py, test_silver_engines.py and 1 more
source_versioned.stamp
function, line 260
source_versioned · stages/history.py:265
snapshot · stages/history.py:224 (value)
no test names it
history
function, line 313
module level · stages/__init__.py:45 (value)gold_helpers.py, silver_helpers.py and 18 more

stages/privacy.py

The privacy scan (B20 C15, SEC-01): restricted values in clear. Pure PySpark (Spark, DuckDB, Databricks). The thirteen stages

DefinitionWhat it does (its own first sentence)Called byNamed in tests
clear_counts
function, line 13
Values present in each column that do not match the token pattern.scan · silver/privacy_scan.py:28test_privacy_stage.py

stages/rows.py

Stages 10, 11 and 13: duplicates, keys, provenance. Pure functions over DataFrames (ADR-013). The thirteen stages

DefinitionWhat it does (its own first sentence)Called byNamed in tests
recent
function, line 22
Silver rows inside the dataset's late lookback (by bronze window), or all of silver when it has none.duplicates · stages/rows.py:56test_runtime_sql_views.py
duplicates
function, line 31
Within the batch: business_key_first keeps the first arrival per key, business_key_latest the latest by the order column, full_row drops identical rows. A type 2 history keeps every distinct change (the same key and order value twice is onemodule level · stages/__init__.py:43 (value)test_contract_rules.py, test_gold_engine.py and 6 more
keys
function, line 64
Every business key column present, else the row is quarantined; the key hash every later join uses.module level · stages/__init__.py:44 (value)
duplicates · stages/rows.py:57 (value)
no test names it
provenance
function, line 72
Where each row came from and when silver took it: batch, file, bronze window and run, silver run, load time, and a record hash over the business columns. Columns come out in a fixed order: contract, other source columns, then load, transpormodule level · stages/__init__.py:46 (value)test_silver_declarative.py, test_silver_stages.py and 1 more