Data PlatformZoom level 3
Who calls what: silver
Every class, function and method in the silver package, with who calls it. Found by reading the code. A call written as object.method() is matched by the method's name, so a same-named method elsewhere can appear: those rows say by name.
83 definitions in 11 modules. Jump to a module: baseline_writer.py · contract_rules.py · daybyday.py · declarative.py · engines.py · history_contract.py · live_recon.py · plan.py · ports.py · privacy_scan.py · runner.py
silver/baseline_writer.py
The baseline batch writer (B8, assumed decision A3): compact bronze up to the cut-off, write it back to bronze as one batch, replay it through the processors and compare with silver; log it only when it matches. Bronze to silver
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
logicalfunction, line 25 | A frame column type as the bronze writer's logical type (decimals as numbers, timestamps with zones too). | SilverBaseliner.write · silver/baseline_writer.py:64 | gold_helpers.py, test_silver_plan.py and 2 more |
baseline_keyfunction, line 35 | SilverBaseliner.write · silver/baseline_writer.py:62 | no test names it | |
SilverBaselinerclass, line 40 | main.housekeeping · scripts/run_platform.py:500 | no test names it | |
SilverBaseliner.writemethod, line 46 | none found in the code | gold_helpers.py, test_cloud_adapters.py and 31 more | |
SilverBaseliner._checkmethod, line 80 | SilverBaseliner.write · silver/baseline_writer.py:67 | no test names it | |
prove_rebuildfunction, line 97 | Rebuild these datasets' silver from what bronze still holds (ADR-025: baseline + later batches) and compare every current-row measure before and after. Returns, per dataset, the measures that differ (empty = proven). | main.prove_rebuild_after · scripts/run_platform.py:526 | no test names it |
prove_rebuild.measuresfunction, line 103 | prove_rebuild · silver/baseline_writer.py:111 | test_semantic_governance.py, gold_helpers.py and 17 more |
silver/contract_rules.py
The contract's row rules as quality-stage rules (DAC DQ-01: rules written in the contract run; ADR-009: severity sets the action). Pure and product-free; the predicates are the engine-neutral SQL subset both engines take. Bronze to silver
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
_quotefunction, line 30 | translate · silver/contract_rules.py:45 | no test names it | |
translatefunction, line 34 | The quality-stage rule, or the reason it is not one. | contract_rules · silver/contract_rules.py:60 | test_contract_rules.py |
contract_rulesfunction, line 58 | (row rules for the quality stage, reasons for every rule that is not one). | module level · realms/databricks/declarative/silver.py:81SilverBaseliner.write · silver/baseline_writer.py:56_one_leased · silver/runner.py:148 | test_contract_rules.py |
with_overridesfunction, line 64 | The dataset's own severity -> action choices (manifest `quality.actions`, ADR-009) laid over the rules the store gave: a dataset may flag what the contract would quarantine, or the other way round. | _one_leased · silver/runner.py:149 | test_contract_rules.py |
silver/daybyday.py
Silver per world day: after the day's ingestion, one silver run takes the new bronze batches, and silver is compared with the oracle's silver totals for the day (current rows, versions, deletes marked, and the manifest's reconciliation meas Bronze to silver
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
SilverDayclass, line 35 | judge_silver · silver/daybyday.py:196 | test_silver_runner.py | |
SilverDay.differencesmethod, line 44 | show_silver · scripts/run_platform.py:250 (value)show_gold · scripts/run_platform.py:267 (value) | test_end_to_end.py, test_explain.py and 1 more | |
SilverDay.okmethod, line 52 | write_ops_page · scripts/run_platform.py:134 (value)success_scorecard · scripts/run_platform.py:182 (value) | no test names it | |
plansfunction, line 56 | judge_silver · silver/daybyday.py:195 | test_gold_plan.py, test_silver_runner.py | |
_hoursfunction, line 64 | hourly · silver/daybyday.py:72 | no test names it | |
hourlyfunction, line 70 | Silver rows against the oracle's, per event hour (the oracle's hours only). | compare · silver/daybyday.py:88 | no test names it |
comparefunction, line 78 | Every number the oracle gives for silver that silver can count, per dataset, as (silver, oracle). | judge_silver · silver/daybyday.py:196 | test_profiling.py, test_silver_stages.py and 4 more |
_stored_keyfunction, line 94 | A business key as silver stores it: a restricted key column holds the token, so the world's clear value is tokenised the way ingestion did before it is looked up. | judge · silver/daybyday.py:117 | no test names it |
judgefunction, line 104 | judge_silver · silver/daybyday.py:208 | test_end_to_end.py, test_explain.py and 3 more | |
judge.vfunction, line 113 | judge · silver/daybyday.py:119SilverDay.differences · silver/daybyday.py:48 (value)compare · silver/daybyday.py:86 (value)_stored_key · silver/daybyday.py:99 (value) | no test names it | |
_tracked_datefunction, line 156 | The column the finding names the current value of (current_<column>). | judge · silver/daybyday.py:133 | no test names it |
silver_dayfunction, line 161 | One silver run for the day, then the live source against silver (REC-05, for datasets `live` can read), silver against the oracle and the day's silver incidents. | main.on_day · scripts/run_platform.py:585 | test_silver_world.py, test_twin_engine.py |
judge_silverfunction, line 181 | The day's silver run judged from its report: the live source against silver, silver against the oracle and the day's silver incidents. The run itself may have happened in another process (a deployed silver job). | main.on_day · scripts/run_platform.py:582silver_day · silver/daybyday.py:178 | test_slice1_local.py |
silver/declarative.py
The declarative silver engine's recipe for one dataset (ADR-035, DAC ENG-02). Pure and product-free. Bronze to silver
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
ChangeCaptureclass, line 22 | recipe · silver/declarative.py:52 | test_silver_declarative.py | |
Recipeclass, line 32 | recipe · silver/declarative.py:60 | no test names it | |
recipefunction, line 40 | module level · realms/databricks/declarative/silver.py:82 | test_silver_declarative.py, test_local_delta_writer.py and 1 more |
silver/engines.py
The bronze-to-silver engines and what each can honour (ADR-035, DAC ENG-02). Bronze to silver
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
EngineCapsclass, line 22 | module level · silver/engines.py:28module level · silver/engines.py:27 (value)engine_problems · silver/engines.py:47 (value) | test_silver_engines.py | |
engine_offunction, line 35 | module level · realms/databricks/declarative/silver.py:78engine_problems · silver/engines.py:50 | test_silver_engines.py | |
history_typefunction, line 39 | engine_problems · silver/engines.py:54 | test_silver_engines.py | |
engine_problemsfunction, line 46 | Why this dataset cannot run on its chosen engine; empty when it can. | compile_domain · meta/loader.py:143 | test_silver_engines.py |
silver/history_contract.py
The SCD Type 2 contract every bronze-to-silver engine must pass (ADR-035, DAC ENG-02). Pure data, product-free. Bronze to silver
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
slugfunction, line 49 | none found in the code | no test names it | |
normalise_closedfunction, line 53 | An engine that closes a deleted key's last version (end set, nothing after it): the last version of a key is 'current'; it is 'deleted' when it is closed. | none found in the code | no test names it |
silver/live_recon.py
Live source against silver (DPC-0052, DAC REC-05): a source can lose rows silently (a change feed that skips changes during maintenance), so every day the live table is compared with silver's current rows. Pure and product-free. Bronze to silver
silver/plan.py
The silver plan: what the thirteen standard stages (ADR-007) do for one dataset, read from the metadata store's merged view of the dataset (manifest + contract columns). Pure and product-free (R1): no dataframe here, only settings. Bronze to silver
silver/ports.py
The ports silver talks to (data-platform-catalog dpc/ports.yaml: stage_runner, and table_writer for silver). Bronze to silver
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
StageResultclass, line 18 | What one run of the stages gives back (stage_runner operation run). | DuckStageRunner.run · adapters/duckdb_silver.py:87SparkStageRunner.run · adapters/spark_silver.py:124 | test_silver_runner.py |
StageRunnerclass, line 27 | stage_runner: the thirteen standard stages, in order, on one dataset's new bronze batches. | none found in the code | no test names it |
StageRunner.runmethod, line 30 | Run every stage. `current` is silver as it stands (None before the first write). No reads or writes happen here: the frames come in and go out (ADR-013). A batch that breaks its contract raises StageError. | none found in the code | no test names it |
StageRunner.releasemethod, line 34 | Free what the run held (cached frames) once the result is written. | none found in the code | test_meta_store.py, test_silver_ports.py and 2 more |
SilverTablesclass, line 38 | The read and write edges of silver: bronze batches in, silver and quarantine out. | none found in the code | no test names it |
SilverTables.bronzemethod, line 41 | The bronze rows of these batches. | SilverBaseliner.write · silver/baseline_writer.py:63 (by name)SilverBaseliner._check · silver/baseline_writer.py:81 (by name)scan · silver/privacy_scan.py:32 (by name)_one_leased · silver/runner.py:171 (by name) | test_access_model.py, test_baseline_stage.py and 45 more |
SilverTables.currentmethod, line 44 | Silver as it stands, or None if the table was never written. | history · stages/history.py:314 (value)duplicates · stages/rows.py:56 (value)DeltaRsSilverTables.count_by_hour · adapters/duckdb_silver.py:127 (by name)SilverBaseliner._check · silver/baseline_writer.py:86 (by name)scan · silver/privacy_scan.py:26 (by name)_one_leased · silver/runner.py:171 (by name) | gold_helpers.py, silver_helpers.py and 14 more |
SilverTables.resetmethod, line 47 | Remove the dataset's silver table and its quarantine, before a rebuild from bronze (ADR-025). | none found in the code | test_governance_catalog.py, test_meta_store.py and 7 more |
SilverTables.writemethod, line 50 | Write the result in one transaction per table: append for append kinds, otherwise a merge on the row key that applies upserts and removes stale versions. Writing the same result twice changes nothing. Quarantined rows go to the dataset's qu | none found in the code | gold_helpers.py, test_cloud_adapters.py and 31 more |
SilverTables.measuresmethod, line 55 | Current rows (current version, not deleted): row_count and each measure; plus versions (every row) and deleted_marked (current rows marked deleted). A table never written gives zeros. | lint · obs/views.py:121 (value)prove_rebuild.measures · silver/baseline_writer.py:108 (value)compare · silver/daybyday.py:85 (value)run_gate · twin/gate.py:202 (value)prove_rebuild.measures · silver/baseline_writer.py:108 (by name)compare · silver/daybyday.py:85 (by name) | test_semantic_governance.py, gold_helpers.py and 17 more |
SilverTables.current_keysmethod, line 59 | The business keys of the current, not deleted rows (the mass-delete guard's comparison, DPC-0054), or None if the table was never written. | _guard_keys · ingest/pipelines.py:239 (by name) | test_ingest_pipelines.py, test_silver_ports.py |
SilverTables.bucket_totalsmethod, line 64 | Totals per business-key bucket over current, not deleted rows (dataplatform.reconcile.bucket_of). | reconcile_live · silver/live_recon.py:60 (by name) | test_live_recon.py, test_reconcile_buckets.py and 1 more |
SilverTables.keys_in_bucketsmethod, line 68 | The business keys, as text, of current, not deleted rows in the wanted buckets only. | reconcile_live · silver/live_recon.py:68 (by name) | test_live_recon.py |
SilverTables.versionsmethod, line 71 | Every row silver holds for one business key, oldest version first (for checks). | UcIcebergStore._create · adapters/table_store.py:399 (value)judge · silver/daybyday.py:117 (by name) | silver_helpers.py, test_gold_harness.py and 5 more |
SilverTables.count_by_hourmethod, line 74 | Current rows per hour of a timestamp column in [start, end), hours counted from `start` (ISO, UTC; a business day may start on the half hour): {hour start ISO: rows}. | hourly · silver/daybyday.py:73 (by name) | test_silver_ports.py |
StageErrorclass, line 79 | A batch the stages cannot take: it stops (DPC-0008), the dataset run fails, nothing is written. | DuckStageRunner.run · adapters/duckdb_silver.py:73SparkStageRunner.run · adapters/spark_silver.py:109DeltaSilverTables.bronze · adapters/spark_silver.py:164StoreSilverTables.bronze · adapters/spark_silver.py:295contract_check · stages/clean.py:49project · stages/clean.py:60and 3 more | test_silver_ports.py, test_silver_runner.py and 1 more |
silver/privacy_scan.py
Run the privacy scan over every enabled dataset's bronze and silver (B20 C15). Engine-free: frames come through the silver ports. Returns (layer, dataset.column, values in clear) per restricted column. Bronze to silver
| Definition | What it does (its own first sentence) | Called by | Named in tests |
|---|---|---|---|
scanfunction, line 15 | success_scorecard · scripts/run_platform.py:191 | no test names it |
silver/runner.py
The silver run: for each enabled dataset, the bronze batches loaded since silver last ran go through the thirteen stages and into silver, and every step is recorded in the metadata store under the run id (ADR-010). Bronze to silver