Data PlatformZoom level 3
Who calls what: stages
Every class, function and method in the stages package, with who calls it. Found by reading the code. A call written as object.method() is matched by the method's name, so a same-named method elsewhere can appear: those rows say by name.
54 definitions in 7 modules. Jump to a module: __init__.py · baseline.py · clean.py · common.py · history.py · privacy.py · rows.py
stages/__init__.py
The thirteen standard bronze-to-silver stages (ADR-007) as pure PySpark functions: DataFrames in, DataFrames out, no reads or writes (ADR-013). This is the in-house part of the stage_runner port the engine binding names (dbs/engine/databric The thirteen stages
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
run_stagesfunction, line 53 | Every stage in order; a stage the plan switches off is skipped. Returns the rows to write, the quarantined rows of every stage, and the frames to count, named <stage>.<what>. `hold`, when the engine adapter passes one, fixes each stage's ou | declare.staged · realms/databricks/declarative/silver.py:42DuckStageRunner.run · adapters/duckdb_silver.py:67SparkStageRunner.run · adapters/spark_silver.py:103 | test_baseline_stage.py, test_silver_stages.py |
stages/baseline.py
The baseline batch (B8, ADR-029, assumed decision A3): a compaction of bronze, not a copy of silver. The thirteen stages
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
compactablefunction, line 27 | '' when the dataset can have a baseline, else the reason it keeps its bronze. | SilverBaseliner.write · silver/baseline_writer.py:53compact · stages/baseline.py:47 | test_baseline_stage.py |
_bronze_namefunction, line 41 | compact · stages/baseline.py:50 | no test names it | |
compactfunction, line 45 | The last row per business key (history order first, then arrival), in bronze's own columns. | SilverBaseliner.write · silver/baseline_writer.py:63 | test_baseline_stage.py |
_livefunction, line 58 | replay_check · stages/baseline.py:72 | no test names it | |
replay_checkfunction, line 68 | Live keys of the replayed baseline against live keys of current silver, leaving out keys any later batch touched (their silver rows moved on after the cut-off). Every frame carries _key_hash (the keys processor). Returns baseline_keys, silv | SilverBaseliner._check · silver/baseline_writer.py:86 | test_baseline_stage.py |
stages/clean.py
Stages 1-9: contract check, projection, hygiene, parse and type, time, standardise, personal data, row filter, quality. Each is a pure function over a DataFrame (ADR-013); the settings come from the plan (ADR-007). The thirteen stages
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
_source_namefunction, line 37 | contract_check · stages/clean.py:45project · stages/clean.py:59 | no test names it | |
contract_checkfunction, line 42 | Every contract column the plan keeps is in the batch (DPC-0008). A missing one stops the batch, naming it. | module level · stages/__init__.py:34 (value) | test_silver_stages.py |
projectfunction, line 54 | All columns by default (ADR-008): contract columns under their contract names (renamed or extracted from the held columns), every other source column too unless the projection is 'declared'; excluded columns go. | module level · stages/__init__.py:35 (value) | test_cloud_adapters.py, test_lakebase.py and 6 more |
clean_textfunction, line 73 | LakeIngestionFramework's default-on hygiene processors, as one column expression: byte-order mark, control characters, line endings, leading and trailing whitespace, runs of spaces, and empty text to null. | hygiene · stages/clean.py:89 | no test names it |
hygienefunction, line 84 | module level · stages/__init__.py:36 (value) | test_silver_declarative.py, test_silver_plan.py and 6 more | |
_try_castfunction, line 94 | CAST that gives null when the value does not convert, on every engine (Spark with ANSI on or off, DuckDB): try_cast. The column name is checked and quoted; the type is one of SQL_TYPES, never data. | cast_to · stages/clean.py:114 | no test names it |
cast_tofunction, line 106 | The contract type, or null when the value does not convert. A fraction never becomes a whole number. try_cast and try_to_timestamp: a value that does not convert is null, never an error, whatever the engine. | parse_type · stages/clean.py:135 | no test names it |
parse_typefunction, line 124 | Cast each column to its contract type. A value that does not convert is quarantined, or emptied, flagged and counted when its column says on_cast_failure: empty (DQ-02): never silently emptied. | module level · stages/__init__.py:37 (value) | test_silver_plan.py, test_silver_stages.py |
timefunction, line 151 | Timestamps stay instants in UTC (the session's zone). A column with local_tz gets <name>_local beside it: the wall-clock time in that zone, as text (ISO, no offset). | module level · stages/__init__.py:38 (value) | no test names it |
standardisefunction, line 167 | Declared value maps (a value not in the map is kept as it is) and letter case. | module level · stages/__init__.py:39 (value) | test_silver_stages.py |
personal_datafunction, line 187 | A restricted column must hold tokens, never the plain value (DPC-0070): a row with a plain value is quarantined. The token format comes from the binding's tokenizer. | module level · stages/__init__.py:40 (value) | test_silver_stages.py |
row_filterfunction, line 204 | Declared filters: rows that do not satisfy one are dropped and counted (not quarantined: they are not wrong). | module level · stages/__init__.py:41 (value) | test_silver_stages.py |
qualityfunction, line 216 | Row rules; severity sets the action (ADR-009): quarantine the row, or flag it in _dq_flags and keep it. | module level · stages/__init__.py:42 (value) | silver_helpers.py, test_access_model.py and 12 more |
stages/common.py
What every stage shares: its inputs, its output, and the quarantine record shape. The thirteen stages
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
StageCtxclass, line 23 | run_stages · stages/__init__.py:66module level · stages/common.py:38 (value) | test_silver_stages.py | |
Outclass, line 32 | contract_check · stages/clean.py:50project · stages/clean.py:69hygiene · stages/clean.py:90parse_type · stages/clean.py:147time · stages/clean.py:163standardise · stages/clean.py:183and 8 more | no test names it | |
quarantine_rowsfunction, line 41 | Split a frame on `reason` (null = good row): (good rows, quarantine records keeping every original value of the `record` columns, all columns by default). | parse_type · stages/clean.py:141personal_data · stages/clean.py:199quality · stages/clean.py:225source_versioned · stages/history.py:291keys · stages/rows.py:67 | no test names it |
reasonsfunction, line 59 | One text naming every failed check, or null when none failed. | parse_type · stages/clean.py:141personal_data · stages/clean.py:199quality · stages/clean.py:225keys · stages/rows.py:67 | test_contract_rules.py, test_hop_reports.py and 3 more |
null_markfunction, line 68 | The text standing for a null key part in a hash: one NUL character. Built by chr(0) rather than a literal so every engine reads the same character (a NUL inside a SQL string literal does not survive on all of them; docs/DUCKDB-SPIKE.md). Th | text_parts · stages/common.py:77 | test_silver_stages.py |
text_partsfunction, line 75 | The columns as one text, SEP between parts, a null part as null_mark(). | key_hash · stages/common.py:81scd2 · stages/history.py:145 | test_silver_stages.py |
key_hashfunction, line 80 | keys · stages/rows.py:68 | no test names it | |
column_typesfunction, line 84 | Each column's type in Spark's simple form (string, bigint, timestamp, ...), read from the schema (portable: `DataFrame.dtypes` is not on every engine). | hygiene · stages/clean.py:85 | test_obs_dashboards.py, test_ops_export.py |
no_flagsfunction, line 90 | An empty list of flag names. Written as SQL text because an empty F.array() does not compile on every engine the stages run on (docs/DUCKDB-SPIKE.md); Spark reads it the same way. | project · stages/clean.py:68add_flags · stages/common.py:101 | no test names it |
add_flagsfunction, line 96 | Append flag names to _dq_flags where their condition holds. | parse_type · stages/clean.py:142quality · stages/clean.py:226 | no test names it |
arrival_orderfunction, line 107 | How rows arrived, oldest first: feed position, stream sequence, arrival time, file name, batch. | duplicates · stages/rows.py:38 | no test names it |
arrival_descfunction, line 112 | Newest arrival first: the latest delivery of the same change wins a tie. | compact · stages/baseline.py:54scd2 · stages/history.py:158scd1 · stages/history.py:182source_versioned · stages/history.py:273duplicates · stages/rows.py:49 | no test names it |
stages/history.py
Stage 12: history by dataset kind (ADR-016), deletes soft. A pure function: new rows and silver as it stands in, the rows to write out. Every output row has a `_row_key` and an `_action`: `upsert` (write it) or `delete` (a stale row of a ke The thirteen stages
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
workingfunction, line 32 | Silver rows in the stages' working shape (bronze run under its bronze name, silver-only columns gone). | snapshot · stages/history.py:236history · stages/history.py:314 | test_gold_dbt_parse.py |
_tsfunction, line 40 | _incoming · stages/history.py:51append · stages/history.py:213 | no test names it | |
_opfunction, line 46 | _incoming · stages/history.py:51_existing · stages/history.py:59 | no test names it | |
_incomingfunction, line 50 | scd2 · stages/history.py:139scd1 · stages/history.py:180 | no test names it | |
_existingfunction, line 58 | _key_compare_deletes · stages/history.py:87_events · stages/history.py:109 | no test names it | |
_key_compare_deletesfunction, line 69 | A full copy is the whole source: a key live in silver and missing from the copy was deleted at the source (DPC-0017). The delete takes the copy's order value and time. The mass-delete guard has already held any copy missing too many keys (D | _events · stages/history.py:105 | no test names it |
_eventsfunction, line 101 | (every event of the keys this batch touches, silver's rows of those keys). | scd2 · stages/history.py:140scd1 · stages/history.py:180 | no test names it |
_stalefunction, line 112 | Silver rows of the touched keys whose row key the new history no longer has: removed. | scd2 · stages/history.py:174scd1 · stages/history.py:193source_versioned · stages/history.py:306 | no test names it |
_finishfunction, line 120 | scd2 · stages/history.py:174scd1 · stages/history.py:193append · stages/history.py:214snapshot · stages/history.py:235source_versioned · stages/history.py:307 | no test names it | |
_trackedfunction, line 130 | scd2 · stages/history.py:145 | no test names it | |
scd2function, line 137 | module level · stages/history.py:310 (value) | no test names it | |
scd1function, line 178 | module level · stages/history.py:310 (value) | no test names it | |
_livefunction, line 197 | A row that is current, not deleted, with no version history of its own. | append · stages/history.py:214snapshot · stages/history.py:231 | no test names it |
appendfunction, line 209 | module level · stages/history.py:310 (value) | azure_fakes.py, gold_helpers.py and 45 more | |
snapshotfunction, line 218 | module level · stages/history.py:310 (value) | test_meta_loader.py, test_obs_recorder.py and 10 more | |
source_versionedfunction, line 255 | History the source already keeps (ADR-032): every version as delivered, with the source's own validity dates, checked per key and never rebuilt. A key whose versions overlap, leave a gap (unless gaps are allowed) or end before they start is | history · stages/history.py:316 | test_silver_declarative.py, test_silver_engines.py and 1 more |
source_versioned.stampfunction, line 260 | source_versioned · stages/history.py:265snapshot · stages/history.py:224 (value) | no test names it | |
historyfunction, line 313 | module level · stages/__init__.py:45 (value) | gold_helpers.py, silver_helpers.py and 18 more |
stages/privacy.py
The privacy scan (B20 C15, SEC-01): restricted values in clear. Pure PySpark (Spark, DuckDB, Databricks). The thirteen stages
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
clear_countsfunction, line 13 | Values present in each column that do not match the token pattern. | scan · silver/privacy_scan.py:28 | test_privacy_stage.py |
stages/rows.py
Stages 10, 11 and 13: duplicates, keys, provenance. Pure functions over DataFrames (ADR-013). The thirteen stages
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
recentfunction, line 22 | Silver rows inside the dataset's late lookback (by bronze window), or all of silver when it has none. | duplicates · stages/rows.py:56 | test_runtime_sql_views.py |
duplicatesfunction, line 31 | Within the batch: business_key_first keeps the first arrival per key, business_key_latest the latest by the order column, full_row drops identical rows. A type 2 history keeps every distinct change (the same key and order value twice is one | module level · stages/__init__.py:43 (value) | test_contract_rules.py, test_gold_engine.py and 6 more |
keysfunction, line 64 | Every business key column present, else the row is quarantined; the key hash every later join uses. | module level · stages/__init__.py:44 (value)duplicates · stages/rows.py:57 (value) | no test names it |
provenancefunction, line 72 | Where each row came from and when silver took it: batch, file, bronze window and run, silver run, load time, and a record hash over the business columns. Columns come out in a fixed order: contract, other source columns, then load, transpor | module level · stages/__init__.py:46 (value) | test_silver_declarative.py, test_silver_stages.py and 1 more |