cleandataengineering.aiData Platform, from the big picture to the code
You are here: Start here › Code map › Who calls what › gold

Data PlatformZoom level 3

Who calls what: gold

Every class, function and method in the gold package, with who calls it. Found by reading the code. A call written as object.method() is matched by the method's name, so a same-named method elsewhere can appear: those rows say by name.

109 definitions in 8 modules. Jump to a module: calendar.py · daybyday.py · harness.py · inspect.py · keys.py · plan.py · ports.py · windows.py

gold/calendar.py

The date dimension (B15; gold modelling: one shared, role-playing date dimension). Pure and product-free. Silver to gold

DefinitionWhat it does (its own first sentence)Called byNamed in tests
fiscal
function, line 26
(fiscal year, fiscal quarter, fiscal month) for a fiscal year starting on the 1st of start_month; the fiscal year is named by the calendar year it ends in (India: April 2026 to March 2027 is fiscal year 2027).calendar_rows · gold/calendar.py:46test_gold_calendar.py
calendar_rows
function, line 36
One row per day from start to end inclusive. `weekend` holds weekday numbers (Monday = 0).write_calendar · scripts/run_platform.py:203test_gold_calendar.py

gold/daybyday.py

Gold per world day: after the day's silver run, one gold run for the day's business date, then gold is compared with the oracle's gold for the day and the incidents gold owns are judged. Silver to gold

DefinitionWhat it does (its own first sentence)Called byNamed in tests
GoldDay
class, line 24
judge_gold · gold/daybyday.py:65test_world_clinical.py
GoldDay.differences
method, line 33
show_silver · scripts/run_platform.py:250 (value)
show_gold · scripts/run_platform.py:267 (value)
test_end_to_end.py, test_explain.py and 1 more
GoldDay.ok
method, line 38
write_ops_page · scripts/run_platform.py:134 (value)
success_scorecard · scripts/run_platform.py:182 (value)
no test names it
GoldJudge
class, line 42
none found in the codeno test names it
GoldJudge.compare
method, line 43
(gold against the oracle per table and measure, notes kept for later days).none found in the codetest_profiling.py, test_silver_stages.py and 4 more
GoldJudge.judge
method, line 47
Verdicts for the incidents gold owns that the day closes (some are judged days after they are planted).none found in the codetest_end_to_end.py, test_explain.py and 3 more
gold_day
function, line 52
One gold run for the day, then gold against the oracle and the day's gold incidents.main.on_day · scripts/run_platform.py:598test_twin_engine.py
judge_gold
function, line 61
The day's gold run judged from its report; the run itself may have happened in another process.main.on_day · scripts/run_platform.py:595
gold_day · gold/daybyday.py:58
no test names it

gold/harness.py

The gold harness (ADR-003): one run builds the domain's gold tables, dimensions first, then facts. Silver to gold

DefinitionWhat it does (its own first sentence)Called byNamed in tests
Restatement
class, line 39
_fact · gold/harness.py:330
_gold · hop_reports.py:49
test_hop_reports.py
GoldTable
class, line 48
run_gold · gold/harness.py:143
_fail · gold/harness.py:175
_dimension · gold/harness.py:190
_fact · gold/harness.py:343
_gold · hop_reports.py:49
GoldReport.by_name · gold/harness.py:67 (value)
test_hop_reports.py
GoldReport
class, line 61
run_gold · gold/harness.py:156
_gold · hop_reports.py:51
kind_of · hop_reports.py:30 (value)
test_hop_reports.py
GoldReport.by_name
method, line 67
run_days · ingest/daybyday.py:131 (by name)
judge · silver/daybyday.py:148 (by name)
ClinicalGoldJudge._enrolment · world/domains/clinical_gold.py:130 (by name)
ClinicalGoldJudge._later · world/domains/clinical_gold.py:206 (by name)
TyreGoldJudge._sales · world/domains/tyre_gold.py:146 (by name)
TyreGoldJudge._later · world/domains/tyre_gold.py:223 (by name)
test_gold_harness.py, test_hop_reports.py and 11 more
Ctx
class, line 72
run_gold · gold/harness.py:131no test names it
CheckFailed
class, line 83
A check before publish failed: the last good windows stay (ADR-006)._dimension · gold/harness.py:185
_fact · gold/harness.py:295
_key_lookup · gold/harness.py:384
_ledger · gold/harness.py:414
_checks · gold/harness.py:428
run_gold · gold/harness.py:148 (value)
no test names it
plans
function, line 88
Every enabled gold dataset's plan, from the store; dimensions first.run_gold · gold/harness.py:130
reproduce · gold/inspect.py:62
test_gold_plan.py, test_silver_runner.py
silver_watermark
function, line 102
How far an input has loaded in source time (ADR-018): the end of the newest delivery window silver has taken (silver's bookmark counts the file records it took; an empty delivery still moves it)._inputs · gold/harness.py:251no test names it
_time
function, line 111
feed_marks · gold/harness.py:123
_fact · gold/harness.py:321
no test names it
feed_marks
function, line 119
An input fed per feed (one file per dealer): ingestion's per-feed bookmarks, each the feed's newest data time._inputs · gold/harness.py:247no test names it
run_gold
function, line 127
Build every gold table of the domain for business date `as_of` (or only the named tables).run_gold · scripts/run_hop.py:78
gold_day · gold/daybyday.py:57
test_explainer.py, test_gold_harness.py and 1 more
_columns
function, line 159
An input's contract columns as (name, engine type), for an input silver has not written yet._dimension · gold/harness.py:179
_inputs · gold/harness.py:233
no test names it
_step
function, line 166
_dimension · gold/harness.py:182
_trace · gold/harness.py:359
_key_lookup · gold/harness.py:380
_ledger · gold/harness.py:411
_checks · gold/harness.py:425
_record · gold/harness.py:456
no test names it
_fail
function, line 171
run_gold · gold/harness.py:149no test names it
_dimension
function, line 178
run_gold · gold/harness.py:147no test names it
_Inputs
class, line 195
_inputs · gold/harness.py:230no test names it
_active_plans
function, line 206
_inputs · gold/harness.py:234no test names it
_corrections
function, line 211
Active corrections for one input. When a correction moves a row between windows (it corrects the fact's own window column), it is applied only if its plan names both windows: a correction never restates a window by surprise (ADR-017). Those_inputs · gold/harness.py:242no test names it
_inputs
function, line 229
_fact · gold/harness.py:290no test names it
_expected_feeds
function, line 256
The members a per-feed input owes a file for: those of the fact's dimension the feed keys belong to, from their first version in the source (a dealer that joined later owes nothing for earlier months). Without such a dimension, the feeds se_inputs · gold/harness.py:249no test names it
_complete
function, line 270
ADR-018: every input caught up past the window's end (per-feed inputs: every expected feed delivered it). A row the build says still waits for an input (`_input_complete` false) is flagged on its own (finish)._fact · gold/harness.py:304no test names it
_fact
function, line 286
run_gold · gold/harness.py:147no test names it
_numbers
function, line 346
_fact · gold/harness.py:331no test names it
_trace
function, line 354
INSP-01: the exact inputs (silver table and version) and the query this run built from, so one window can be rebuilt later from the run log alone (gold/inspect.py). An unknown version is recorded as -1._inputs · gold/harness.py:252no test names it
_range
function, line 364
The windows this run recomputes: the lookback from as_of, plus the windows active plans name and those with rows on the unknown member; every window when a withdrawn batch's plan says so (VOID-01)._fact · gold/harness.py:299no test names it
_key_lookup
function, line 375
Surrogate keys by the as-of joins (ADR-005), the rows-in = rows-out check and the row ledger (LEDG-01)._fact · gold/harness.py:297no test names it
_which_join
function, line 388
The first dimension join that changed the row count (run only when the total moved): names the step without a rerun by hand (LEDG-01)._key_lookup · gold/harness.py:379no test names it
_ledger
function, line 399
LEDG-01: rows read per input, rows the build gave, and per dimension join rows matched, on the unknown member and not applicable. Each join's three counts must add up to the rows after the lookup, else the run fails._key_lookup · gold/harness.py:382no test names it
_checks
function, line 418
Checks before publish (ADR-006): grain unique (KEY-01), orphan rate (LATE-02), reconcile with what the build gave (no row or quantity lost by the harness). A failure raises: nothing is written._fact · gold/harness.py:309no test names it
_record
function, line 453
_fact · gold/harness.py:336no test names it
summary
function, line 476
The run in plain numbers (for reports).none found in the codetest_bundle.py, test_dac_coverage.py and 1 more

gold/inspect.py

Inspect mode (INSP-01): rebuild one window of a past gold run from what its run log names. Product-free. Silver to gold

DefinitionWhat it does (its own first sentence)Called byNamed in tests
NotInspectable
class, line 28
The run log does not name what is needed (an older run, a dimension, a table the run did not build).reproduce · gold/inspect.py:64no test names it
Reproduced
class, line 33
reproduce · gold/inspect.py:83no test names it
Reproduced.same
method, line 46
none found in the codeno test names it
_plain
function, line 50
Reproduced.same · gold/inspect.py:47no test names it
_steps
function, line 54
reproduce · gold/inspect.py:66no test names it
reproduce
function, line 60
none found in the codetest_gold_harness.py, test_demo_end_to_end.py

gold/keys.py

Deterministic hash surrogate keys with reserved members (ADR-004, DPC-0081, DPC-0046). Landing to bronze

DefinitionWhat it does (its own first sentence)Called byNamed in tests
canonical_time
function, line 26
A version start as the recipe writes it (a naive time is taken as UTC).key_text · gold/keys.py:35no test names it
key_text
function, line 32
surrogate_key · gold/keys.py:42test_gold_plan.py
surrogate_key
function, line 40
The member's key; the same inputs always give the same number.none found in the codetest_gold_engine.py, test_gold_plan.py
member
function, line 46
The key a fact row points at: not applicable without a business key, unknown when no member matched.none found in the codetest_gold_engine.py, test_gold_harness.py and 3 more

gold/plan.py

A gold table's plan, read from its DPM gold manifest (manifest/gold/<name>.yaml) and its ODCS contract. Silver to gold

DefinitionWhat it does (its own first sentence)Called byNamed in tests
GoldPlanError
class, line 24
A gold manifest the harness cannot run (the DPM linter's rules, ADR-003 and ADR-006 confirmations)._range · gold/harness.py:368
plan_from_manifest · gold/plan.py:142
_dim_key · gold/plan.py:206
order · gold/plan.py:216
run_gold · gold/harness.py:148 (value)
test_gold_plan.py
DimRef
class, line 29
A dimension a fact points at: its surrogate key column, the business key in the fact, and the fact column the as-of join reads (ADR-005)._dims · gold/plan.py:116
module level · gold/plan.py:76 (value)
_dims · gold/plan.py:115 (value)
no test names it
Window
class, line 40
plan_from_manifest · gold/plan.py:156test_silver_stages.py, test_dq_types_and_loader.py
Input
class, line 47
plan_from_manifest · gold/plan.py:181
module level · gold/plan.py:75 (value)
no test names it
Input.source
method, line 53
main · realms/aws-databricks/meta_setup.py:88 (value)
module level · realms/databricks/declarative/silver.py:77 (value)
lakebase · realms/databricks/engine/tasks/probe.py:83 (value)
hop_delays · scripts/run_platform.py:161 (value)
AzureLanding.read_bytes · adapters/azure_blob.py:102 (value)
AzureLanding.manifest · adapters/azure_blob.py:105 (value)
and 70 more
test_bundle.py, test_semantic_governance.py and 70 more
Input.name
method, line 57
none found in the codeno test names it
Reconcile
class, line 62
_reconcile · gold/plan.py:124no test names it
GoldPlan
class, line 68
plan_from_manifest · gold/plan.py:174
module level · gold/harness.py:80 (value)
no test names it
GoldPlan.schema
method, line 91
The gold schema (ADR-012): gold_<domain>; locally a folder prefix, on the engine `<env>_gold.<domain>`.intents · realms/aws-databricks/lakeformation.py:41 (value)
terraform · realms/aws-databricks/lakeformation.py:77 (value)
main · realms/databricks/dashboards/generate.py:53 (value)
module level · realms/databricks/engine/tasks/probe.py:23 (value)
catalog_action_sql · realms/databricks/semantic_governance/render.py:268 (value)
_grant_sql · realms/databricks/semantic_governance/render.py:301 (value)
and 22 more
test_lakeformation.py, test_semantic_governance.py and 37 more
GoldPlan.table
method, line 96
none found in the codetest_lakeformation.py, test_semantic_governance.py and 84 more
GoldPlan.column_names
method, line 100
_print_delta_summary · scripts/demo_end_to_end.py:815 (value)test_bronze_to_silver.py, test_demo_end_to_end.py and 4 more
GoldPlan.surrogate
method, line 104
A dimension's own surrogate key column (its grain).SparkGoldEngine.dimension · adapters/spark_gold.py:105 (value)
_dimension · gold/harness.py:181 (value)
SparkGoldEngine.dimension · adapters/spark_gold.py:104 (by name)
test_semantic_governance.py, test_gold_plan.py
GoldPlan.measure_columns
method, line 109
Number columns of a fact that are not keys: what a window's numbers are made of.none found in the codetest_gold_plan.py
_dims
function, line 115
plan_from_manifest · gold/plan.py:165no test names it
_reconcile
function, line 120
plan_from_manifest · gold/plan.py:192no test names it
plan_from_view
function, line 128
The plan from the metadata store's view of a gold dataset (its manifest and contract columns).plans · gold/harness.py:97silver_helpers.py, test_explain.py and 3 more
plan_from_manifest
function, line 136
The plan for one gold manifest. Refuses what the DPM linter refuses (ADR-003, ADR-006). `input_keys` maps an input dataset to its primary key (from the metadata store): a dimension's business key is its input's key.plan_from_view · gold/plan.py:133test_gold_engine.py, test_gold_plan.py and 1 more
_dim_key
function, line 199
plan_from_manifest · gold/plan.py:195no test names it
order
function, line 210
Dimensions first, then facts; each group by name. A fact naming a dimension no plan builds is refused.plans · gold/harness.py:98azure_fakes.py, gold_helpers.py and 31 more

gold/ports.py

The ports gold talks to (dpc/ports.yaml: table_writer for gold, and the engine operations the harness asks for). Silver to gold

DefinitionWhat it does (its own first sentence)Called byNamed in tests
GoldTables
class, line 20
The read and write edges of gold.none found in the codetest_gold_ports.py
GoldTables.silver
method, line 23
Every row silver holds for `source.dataset` (every version, deleted ones marked). A table never written (an input that has delivered nothing yet) is an empty frame of its contract `columns` (name, engine type) and silver's own columns: an i_dimension · gold/harness.py:179 (by name)
_inputs · gold/harness.py:233 (by name)
reproduce · gold/inspect.py:76 (by name)
test_bundle.py, test_semantic_governance.py and 62 more
GoldTables.silver_version
method, line 28
The silver table's current version (commit or snapshot), recorded per gold run (INSP-01); None when the table was never written or the format cannot say._inputs · gold/harness.py:232 (by name)no test names it
GoldTables.silver_at
method, line 32
The silver table as it was at `version` (time travel), to rebuild a past run's window (INSP-01).reproduce · gold/inspect.py:76 (by name)no test names it
GoldTables.gold
method, line 35
The gold table as published, or None before its first write.build · realms/databricks/semantic_governance/generate.py:110 (value)
main · scripts/run_platform.py:459 (value)
_expected_feeds · gold/harness.py:261 (by name)
_fact · gold/harness.py:292 (by name)
reproduce · gold/inspect.py:79 (by name)
no test names it
GoldTables.write_dimension
method, line 38
Replace the whole dimension in one transaction (its keys are deterministic, so a rebuild changes nothing that did not change). Returns the rows written._dimension · gold/harness.py:186 (by name)test_gold_ports.py
GoldTables.replace_windows
method, line 42
Replace exactly these windows of a fact with `rows` in one transaction: rows of other windows are never touched, rows of these windows not in `rows` are removed (ADR-006: replace, never append). Writing the same rows twice changes nothing. _fact · gold/harness.py:318 (by name)test_gold_ports.py
GoldEngine
class, line 48
Frame operations of the harness, pure: frames in, frames or numbers out.none found in the codeno test names it
GoldEngine.dimension
method, line 51
The common one-to-one pipeline: contract columns, hash surrogate keys per version (ADR-004), valid from and to (the first version from the beginning of time so an early fact finds it, the current one to the end of time), and the reserved me_where · realms/databricks/semantic_governance/render.py:131 (value)
_tile_query · realms/databricks/semantic_governance/render.py:169 (value)
_widget · realms/databricks/semantic_governance/render.py:191 (value)
scorecard · obs/success.py:157 (value)
_where · semantic/answers.py:21 (value)
_dimension · gold/harness.py:180 (by name)
and 1 more
test_semantic_governance.py, gold_helpers.py and 10 more
GoldEngine.overlay
method, line 56
Lay active corrections over silver rows (ADR-017): each correction's field takes its value on every row of its business key. Silver itself is never changed._inputs · gold/harness.py:245 (by name)test_gold_engine.py
GoldEngine.query_text
method, line 60
What the build runs, for people (INSP-01): the SQL model with its parameter values shown, or the name of the dataframe function. Never executed: the build binds the parameters itself._trace · gold/harness.py:360 (by name)test_lineage_end_to_end.py, test_adf_generator.py and 1 more
GoldEngine.build
method, line 64
Call the fact's declared build() (SQL model or dataframe function) on its input frames.cmd_code · scripts/explain.py:101 (by name)
_fact · gold/harness.py:291 (by name)
reproduce · gold/inspect.py:78 (by name)
Monitor.tables · obs/monitor.py:122 (by name)
test_semantic_governance.py, conftest.py and 17 more
GoldEngine.add_keys
method, line 67
Each dimension's surrogate key by a left join as of the event (ADR-005): valid_from <= t < valid_to; no member -> -1, no business key -> -2. Never drops or multiplies a row._key_lookup · gold/harness.py:377 (by name)
_which_join · gold/harness.py:392 (by name)
reproduce · gold/inspect.py:80 (by name)
test_gold_engine.py
GoldEngine.finish
method, line 71
Harness columns (_is_complete: the window's flag, and false on a row whose `_input_complete` is false; _run_id) and the contract's column order.none found in the codetest_gold_engine.py, test_meta_store.py and 2 more
GoldEngine.only_windows
method, line 75
The rows of windows from `first` on, plus the windows named in `also`._fact · gold/harness.py:307 (by name)
_checks · gold/harness.py:443 (by name)
reproduce · gold/inspect.py:80 (by name)
no test names it
GoldEngine.window_numbers
method, line 78
Per window from `first` on (plus `also`): its numbers and a digest of every row's content but _run_id._fact · gold/harness.py:300 (by name)
reproduce · gold/inspect.py:81 (by name)
test_gold_engine.py
GoldEngine.hold
method, line 82
The same rows, computed once and kept for the operations that follow (released with `release`)._fact · gold/harness.py:296 (by name)
_key_lookup · gold/harness.py:377 (by name)
no test names it
GoldEngine.count
method, line 85
none found in the codetest_contract_rules.py, test_dac_coverage.py and 29 more
GoldEngine.duplicates
method, line 87
Rows beyond the first per value of `columns` (0 when unique)._dimension · gold/harness.py:181 (by name)
_checks · gold/harness.py:424 (by name)
test_contract_rules.py, test_gold_engine.py and 6 more
GoldEngine.key_tally
method, line 90
Per dimension key: rows matched to a member, on the unknown member (-1), not applicable (-2) (LEDG-01)._ledger · gold/harness.py:406 (by name)no test names it
GoldEngine.unknown_rows
method, line 93
Rows on the unknown member, per dimension key (orphans, LATE-02)._checks · gold/harness.py:429 (by name)test_gold_engine.py
GoldEngine.windows_with_unknown
method, line 96
Published windows still holding rows on the unknown member: re-keyed on the next recompute (ADR-004)._range · gold/harness.py:372 (by name)no test names it
GoldEngine.rows_incomplete
method, line 99
Windows where a build said a row's inputs have not all arrived (its `_input_complete` column is false)._fact · gold/harness.py:301 (by name)no test names it
GoldEngine.first_seen
method, line 102
Each real member's first version start in its source (for which feeds a window expects)._expected_feeds · gold/harness.py:264 (by name)no test names it
GoldEngine.totals
method, line 105
Sum of each column (for reconciliation)._checks · gold/harness.py:443 (by name)test_ingest_world.py, test_reconcile_engine.py and 3 more
GoldEngine.release
method, line 108
Free what the operations held for these frames (cached rows) once the table is written.none found in the codetest_meta_store.py, test_silver_ports.py and 2 more
GoldReader
class, line 112
Read-only questions to published gold (checks, the oracle comparison, known answers): one SQL query over the gold tables, each visible by its table name (fact_otif, dim_dealer, ...). Rows come back as plain values.none found in the codetest_gold_ports.py, test_world_clinical.py
GoldReader.rows
method, line 116
`time_zone`: the session time zone the query runs in (default: the engine's, UTC).none found in the codeno test names it

gold/windows.py

Windows of a fact (ADR-006), when one is complete (ADR-018) and when a rebuild is a restatement (ADR-017). Pure. Silver to gold

DefinitionWhat it does (its own first sentence)Called byNamed in tests
_month_add
function, line 20
end · gold/windows.py:37
range_start · gold/windows.py:53
no test names it
start
function, line 25
end · gold/windows.py:35silver_helpers.py, test_gold_calendar.py and 15 more
end
function, line 32
The first instant after the window.day_end · gold/windows.py:62
caught_up · gold/windows.py:67
no test names it
of
function, line 41
The window an instant falls in.range_start · gold/windows.py:57
feeds_caught_up · gold/windows.py:74
no test names it
range_start
function, line 49
The first window the recompute rebuilds on business date `as_of`: `lookback` units back (ADR-006).none found in the codetest_gold_plan.py
day_end
function, line 60
How far an input has loaded in source time once a business day's delivery window is in silver.none found in the codetest_gold_plan.py
caught_up
function, line 65
One input has loaded past the window's end (ADR-018). No watermark: it has loaded nothing yet.none found in the codetest_gold_plan.py
feeds_caught_up
function, line 70
A per-feed input (one file per dealer): each expected feed's newest data time falls in this window or a later one. Returns (complete, feeds still missing).none found in the codetest_gold_plan.py
Numbers
class, line 79
What readers of a window see: its rows and the sums of its measures, and whether it was marked complete.SparkGoldEngine.window_numbers · adapters/spark_gold.py:236
SparkGoldEngine.window_numbers · adapters/spark_gold.py:225 (value)
GoldEngine.window_numbers · gold/ports.py:79 (value)
test_gold_plan.py
Numbers.same_numbers
method, line 86
is_restatement · gold/windows.py:105 (by name)no test names it
is_restatement
function, line 91
A rebuild restates a window when numbers readers were entitled to treat as final change: - the window was published complete and its numbers change (a correction, a late change), or - the window, already over when it was published, was flagnone found in the codetest_gold_plan.py