cleandataengineering.aiData Platform, from the big picture to the code
You are here: Start here › Code map › Who calls what › silver

Data PlatformZoom level 3

Who calls what: silver

Every class, function and method in the silver package, with who calls it. Found by reading the code. A call written as object.method() is matched by the method's name, so a same-named method elsewhere can appear: those rows say by name.

83 definitions in 11 modules. Jump to a module: baseline_writer.py · contract_rules.py · daybyday.py · declarative.py · engines.py · history_contract.py · live_recon.py · plan.py · ports.py · privacy_scan.py · runner.py

silver/baseline_writer.py

The baseline batch writer (B8, assumed decision A3): compact bronze up to the cut-off, write it back to bronze as one batch, replay it through the processors and compare with silver; log it only when it matches. Bronze to silver

DefinitionWhat it does (its own first sentence)Called byNamed in tests
logical
function, line 25
A frame column type as the bronze writer's logical type (decimals as numbers, timestamps with zones too).SilverBaseliner.write · silver/baseline_writer.py:64gold_helpers.py, test_silver_plan.py and 2 more
baseline_key
function, line 35
SilverBaseliner.write · silver/baseline_writer.py:62no test names it
SilverBaseliner
class, line 40
main.housekeeping · scripts/run_platform.py:500no test names it
SilverBaseliner.write
method, line 46
none found in the codegold_helpers.py, test_cloud_adapters.py and 31 more
SilverBaseliner._check
method, line 80
SilverBaseliner.write · silver/baseline_writer.py:67no test names it
prove_rebuild
function, line 97
Rebuild these datasets' silver from what bronze still holds (ADR-025: baseline + later batches) and compare every current-row measure before and after. Returns, per dataset, the measures that differ (empty = proven).main.prove_rebuild_after · scripts/run_platform.py:526no test names it
prove_rebuild.measures
function, line 103
prove_rebuild · silver/baseline_writer.py:111test_semantic_governance.py, gold_helpers.py and 17 more

silver/contract_rules.py

The contract's row rules as quality-stage rules (DAC DQ-01: rules written in the contract run; ADR-009: severity sets the action). Pure and product-free; the predicates are the engine-neutral SQL subset both engines take. Bronze to silver

DefinitionWhat it does (its own first sentence)Called byNamed in tests
_quote
function, line 30
translate · silver/contract_rules.py:45no test names it
translate
function, line 34
The quality-stage rule, or the reason it is not one.contract_rules · silver/contract_rules.py:60test_contract_rules.py
contract_rules
function, line 58
(row rules for the quality stage, reasons for every rule that is not one).module level · realms/databricks/declarative/silver.py:81
SilverBaseliner.write · silver/baseline_writer.py:56
_one_leased · silver/runner.py:148
test_contract_rules.py
with_overrides
function, line 64
The dataset's own severity -> action choices (manifest `quality.actions`, ADR-009) laid over the rules the store gave: a dataset may flag what the contract would quarantine, or the other way round._one_leased · silver/runner.py:149test_contract_rules.py

silver/daybyday.py

Silver per world day: after the day's ingestion, one silver run takes the new bronze batches, and silver is compared with the oracle's silver totals for the day (current rows, versions, deletes marked, and the manifest's reconciliation meas Bronze to silver

DefinitionWhat it does (its own first sentence)Called byNamed in tests
SilverDay
class, line 35
judge_silver · silver/daybyday.py:196test_silver_runner.py
SilverDay.differences
method, line 44
show_silver · scripts/run_platform.py:250 (value)
show_gold · scripts/run_platform.py:267 (value)
test_end_to_end.py, test_explain.py and 1 more
SilverDay.ok
method, line 52
write_ops_page · scripts/run_platform.py:134 (value)
success_scorecard · scripts/run_platform.py:182 (value)
no test names it
plans
function, line 56
judge_silver · silver/daybyday.py:195test_gold_plan.py, test_silver_runner.py
_hours
function, line 64
hourly · silver/daybyday.py:72no test names it
hourly
function, line 70
Silver rows against the oracle's, per event hour (the oracle's hours only).compare · silver/daybyday.py:88no test names it
compare
function, line 78
Every number the oracle gives for silver that silver can count, per dataset, as (silver, oracle).judge_silver · silver/daybyday.py:196test_profiling.py, test_silver_stages.py and 4 more
_stored_key
function, line 94
A business key as silver stores it: a restricted key column holds the token, so the world's clear value is tokenised the way ingestion did before it is looked up.judge · silver/daybyday.py:117no test names it
judge
function, line 104
judge_silver · silver/daybyday.py:208test_end_to_end.py, test_explain.py and 3 more
judge.v
function, line 113
judge · silver/daybyday.py:119
SilverDay.differences · silver/daybyday.py:48 (value)
compare · silver/daybyday.py:86 (value)
_stored_key · silver/daybyday.py:99 (value)
no test names it
_tracked_date
function, line 156
The column the finding names the current value of (current_<column>).judge · silver/daybyday.py:133no test names it
silver_day
function, line 161
One silver run for the day, then the live source against silver (REC-05, for datasets `live` can read), silver against the oracle and the day's silver incidents.main.on_day · scripts/run_platform.py:585test_silver_world.py, test_twin_engine.py
judge_silver
function, line 181
The day's silver run judged from its report: the live source against silver, silver against the oracle and the day's silver incidents. The run itself may have happened in another process (a deployed silver job).main.on_day · scripts/run_platform.py:582
silver_day · silver/daybyday.py:178
test_slice1_local.py

silver/declarative.py

The declarative silver engine's recipe for one dataset (ADR-035, DAC ENG-02). Pure and product-free. Bronze to silver

silver/engines.py

The bronze-to-silver engines and what each can honour (ADR-035, DAC ENG-02). Bronze to silver

silver/history_contract.py

The SCD Type 2 contract every bronze-to-silver engine must pass (ADR-035, DAC ENG-02). Pure data, product-free. Bronze to silver

DefinitionWhat it does (its own first sentence)Called byNamed in tests
slug
function, line 49
none found in the codeno test names it
normalise_closed
function, line 53
An engine that closes a deleted key's last version (end set, nothing after it): the last version of a key is 'current'; it is 'deleted' when it is closed.none found in the codeno test names it

silver/live_recon.py

Live source against silver (DPC-0052, DAC REC-05): a source can lose rows silently (a change feed that skips changes during maintenance), so every day the live table is compared with silver's current rows. Pure and product-free. Bronze to silver

silver/plan.py

The silver plan: what the thirteen standard stages (ADR-007) do for one dataset, read from the metadata store's merged view of the dataset (manifest + contract columns). Pure and product-free (R1): no dataframe here, only settings. Bronze to silver

DefinitionWhat it does (its own first sentence)Called byNamed in tests
PlanError
class, line 91
A dataset's silver settings the engine cannot run: an unknown setting or stage, a missing key or order.recipe · silver/declarative.py:44
SilverPlan.column · silver/plan.py:171
_column · silver/plan.py:185
_rules · silver/plan.py:220
_validity · silver/plan.py:247
plan_from_view · silver/plan.py:267
and 1 more
test_silver_declarative.py, test_silver_plan.py and 1 more
ColumnPlan
class, line 96
_column · silver/plan.py:196
module level · silver/plan.py:137 (value)
SilverPlan.kept · silver/plan.py:174 (value)
_rules · silver/plan.py:213 (value)
test_world_clinical.py
ColumnPlan.restricted
method, line 112
none found in the codetest_lakeformation.py, test_semantic_governance.py and 11 more
Rule
class, line 117
A row rule: rows must satisfy `predicate` (an engine expression); failing rows get `action`.translate · silver/contract_rules.py:46
_rules · silver/plan.py:222
contract_rules · silver/contract_rules.py:61 (value)
with_overrides · silver/contract_rules.py:64 (value)
module level · silver/plan.py:156 (value)
_rules · silver/plan.py:213 (value)
and 4 more
no test names it
Measure
class, line 127
_measures · silver/plan.py:232
DeltaSilverTables.measures · adapters/spark_silver.py:219 (value)
module level · silver/plan.py:161 (value)
_measures · silver/plan.py:230 (value)
SilverTables.measures · silver/ports.py:55 (value)
test_obs_dashboards.py
SilverPlan
class, line 134
plan_from_view · silver/plan.py:305
plans · silver/daybyday.py:56 (value)
compare · silver/daybyday.py:78 (value)
judge · silver/daybyday.py:105 (value)
silver_helpers.py, test_live_recon.py and 1 more
SilverPlan.table
method, line 164
none found in the codetest_lakeformation.py, test_semantic_governance.py and 84 more
SilverPlan.column
method, line 167
catalog_action_sql · realms/databricks/semantic_governance/render.py:270 (value)
_mask_sql · realms/databricks/semantic_governance/render.py:322 (value)
_row_filter_sql · realms/databricks/semantic_governance/render.py:335 (value)
plan_access · access/model.py:158 (value)
measure_expr · adapters/databricks_ops_views.py:32 (value)
DeltaRsGoldTables.replace_windows · adapters/duckdb_gold.py:127 (value)
and 13 more
test_semantic_governance.py, gold_helpers.py and 60 more
SilverPlan.kept
method, line 174
none found in the codeno test names it
SilverPlan.runs
method, line 177
run_stages · stages/__init__.py:71 (by name)no test names it
_column
function, line 182
plan_from_view · silver/plan.py:268no test names it
_rules
function, line 213
Required columns are present (critical). The contract's other row rules reach this stage with the DQ engine (W1e); the stage applies whatever rules the plan holds.plan_from_view · silver/plan.py:327no test names it
_measures
function, line 228
plan_from_view · silver/plan.py:334no test names it
_validity
function, line 240
(valid_from column, valid_to column, gaps allowed) for history the source keeps (ADR-032): from the manifest's history block, else the dataset settings. source_versioned needs both columns, and both must be contract columns.plan_from_view · silver/plan.py:288no test names it
plan_from_view
function, line 253
The plan for one dataset from its merged view (MetadataStore.dataset_view). `token_pattern` is the binding's tokenizer format; `projection` 'declared' keeps only the contract's columns (ADR-008: DPM has no key for it yet).module level · realms/databricks/declarative/silver.py:80
SilverBaseliner.write · silver/baseline_writer.py:50
prove_rebuild.measures · silver/baseline_writer.py:107
plans · silver/daybyday.py:58
scan · silver/privacy_scan.py:21
_one_leased · silver/runner.py:145
silver_helpers.py, test_explain.py and 3 more
check_stage_order
function, line 338
An engine must supply exactly the standard stages, in the standard order (DPC-0045, CFG-01).module level · stages/__init__.py:48test_silver_plan.py

silver/ports.py

The ports silver talks to (data-platform-catalog dpc/ports.yaml: stage_runner, and table_writer for silver). Bronze to silver

DefinitionWhat it does (its own first sentence)Called byNamed in tests
StageResult
class, line 18
What one run of the stages gives back (stage_runner operation run).DuckStageRunner.run · adapters/duckdb_silver.py:87
SparkStageRunner.run · adapters/spark_silver.py:124
test_silver_runner.py
StageRunner
class, line 27
stage_runner: the thirteen standard stages, in order, on one dataset's new bronze batches.none found in the codeno test names it
StageRunner.run
method, line 30
Run every stage. `current` is silver as it stands (None before the first write). No reads or writes happen here: the frames come in and go out (ADR-013). A batch that breaks its contract raises StageError.none found in the codeno test names it
StageRunner.release
method, line 34
Free what the run held (cached frames) once the result is written.none found in the codetest_meta_store.py, test_silver_ports.py and 2 more
SilverTables
class, line 38
The read and write edges of silver: bronze batches in, silver and quarantine out.none found in the codeno test names it
SilverTables.bronze
method, line 41
The bronze rows of these batches.SilverBaseliner.write · silver/baseline_writer.py:63 (by name)
SilverBaseliner._check · silver/baseline_writer.py:81 (by name)
scan · silver/privacy_scan.py:32 (by name)
_one_leased · silver/runner.py:171 (by name)
test_access_model.py, test_baseline_stage.py and 45 more
SilverTables.current
method, line 44
Silver as it stands, or None if the table was never written.history · stages/history.py:314 (value)
duplicates · stages/rows.py:56 (value)
DeltaRsSilverTables.count_by_hour · adapters/duckdb_silver.py:127 (by name)
SilverBaseliner._check · silver/baseline_writer.py:86 (by name)
scan · silver/privacy_scan.py:26 (by name)
_one_leased · silver/runner.py:171 (by name)
gold_helpers.py, silver_helpers.py and 14 more
SilverTables.reset
method, line 47
Remove the dataset's silver table and its quarantine, before a rebuild from bronze (ADR-025).none found in the codetest_governance_catalog.py, test_meta_store.py and 7 more
SilverTables.write
method, line 50
Write the result in one transaction per table: append for append kinds, otherwise a merge on the row key that applies upserts and removes stale versions. Writing the same result twice changes nothing. Quarantined rows go to the dataset's qunone found in the codegold_helpers.py, test_cloud_adapters.py and 31 more
SilverTables.measures
method, line 55
Current rows (current version, not deleted): row_count and each measure; plus versions (every row) and deleted_marked (current rows marked deleted). A table never written gives zeros.lint · obs/views.py:121 (value)
prove_rebuild.measures · silver/baseline_writer.py:108 (value)
compare · silver/daybyday.py:85 (value)
run_gate · twin/gate.py:202 (value)
prove_rebuild.measures · silver/baseline_writer.py:108 (by name)
compare · silver/daybyday.py:85 (by name)
test_semantic_governance.py, gold_helpers.py and 17 more
SilverTables.current_keys
method, line 59
The business keys of the current, not deleted rows (the mass-delete guard's comparison, DPC-0054), or None if the table was never written._guard_keys · ingest/pipelines.py:239 (by name)test_ingest_pipelines.py, test_silver_ports.py
SilverTables.bucket_totals
method, line 64
Totals per business-key bucket over current, not deleted rows (dataplatform.reconcile.bucket_of).reconcile_live · silver/live_recon.py:60 (by name)test_live_recon.py, test_reconcile_buckets.py and 1 more
SilverTables.keys_in_buckets
method, line 68
The business keys, as text, of current, not deleted rows in the wanted buckets only.reconcile_live · silver/live_recon.py:68 (by name)test_live_recon.py
SilverTables.versions
method, line 71
Every row silver holds for one business key, oldest version first (for checks).UcIcebergStore._create · adapters/table_store.py:399 (value)
judge · silver/daybyday.py:117 (by name)
silver_helpers.py, test_gold_harness.py and 5 more
SilverTables.count_by_hour
method, line 74
Current rows per hour of a timestamp column in [start, end), hours counted from `start` (ISO, UTC; a business day may start on the half hour): {hour start ISO: rows}.hourly · silver/daybyday.py:73 (by name)test_silver_ports.py
StageError
class, line 79
A batch the stages cannot take: it stops (DPC-0008), the dataset run fails, nothing is written.DuckStageRunner.run · adapters/duckdb_silver.py:73
SparkStageRunner.run · adapters/spark_silver.py:109
DeltaSilverTables.bronze · adapters/spark_silver.py:164
StoreSilverTables.bronze · adapters/spark_silver.py:295
contract_check · stages/clean.py:49
project · stages/clean.py:60
and 3 more
test_silver_ports.py, test_silver_runner.py and 1 more

silver/privacy_scan.py

Run the privacy scan over every enabled dataset's bronze and silver (B20 C15). Engine-free: frames come through the silver ports. Returns (layer, dataset.column, values in clear) per restricted column. Bronze to silver

DefinitionWhat it does (its own first sentence)Called byNamed in tests
scan
function, line 15
success_scorecard · scripts/run_platform.py:191no test names it

silver/runner.py

The silver run: for each enabled dataset, the bronze batches loaded since silver last ran go through the thirteen stages and into silver, and every step is recorded in the metadata store under the run id (ADR-010). Bronze to silver

DefinitionWhat it does (its own first sentence)Called byNamed in tests
SilverDataset
class, line 30
_silver · hop_reports.py:45
_fail · silver/runner.py:104
_one · silver/runner.py:124
_one_leased · silver/runner.py:163
SilverReport.by_name · silver/runner.py:50 (value)
test_hop_reports.py, test_silver_runner.py
SilverDataset.name
method, line 40
none found in the codeno test names it
SilverReport
class, line 45
_silver · hop_reports.py:45
build_silver · silver/runner.py:97
kind_of · hop_reports.py:28 (value)
test_hop_reports.py, test_silver_runner.py
SilverReport.by_name
method, line 50
run_days · ingest/daybyday.py:131 (by name)
judge · silver/daybyday.py:148 (by name)
ClinicalGoldJudge._enrolment · world/domains/clinical_gold.py:130 (by name)
ClinicalGoldJudge._later · world/domains/clinical_gold.py:206 (by name)
TyreGoldJudge._sales · world/domains/tyre_gold.py:146 (by name)
TyreGoldJudge._later · world/domains/tyre_gold.py:223 (by name)
test_gold_harness.py, test_hop_reports.py and 11 more
new_batches
function, line 54
(batch keys loaded since the bookmark, in file-log order without repeats; the new bookmark). `from_start` takes every loaded batch from the first (a rebuild, ADR-025). A voided batch reads as `void` and is never taken. A baseline batch (houscan · silver/privacy_scan.py:30
_one_leased · silver/runner.py:157
test_housekeeping.py
build_silver
function, line 69
Run silver for every enabled dataset of a domain (or the named `source.dataset`s). `as_of` is the business date of the run (the late lookback counts back from it). Datasets named in `rebuild` (ADR-025, recovery by rerun) lose their silver trun_silver · scripts/run_hop.py:71
prove_rebuild · silver/baseline_writer.py:112
silver_day · silver/daybyday.py:177
test_explainer.py, test_silver_runner.py
_fail
function, line 100
_one_leased · silver/runner.py:147no test names it
_one
function, line 107
One dataset under its silver writer lease (ORCH-04): a second run finds it busy and writes nothing.build_silver · silver/runner.py:93no test names it
_one_leased
function, line 131
_one · silver/runner.py:126no test names it