Data PlatformZoom level 1
Silver to gold: tables shaped for questions
| Part | Starting point | Called by (found by reading the code) |
|---|---|---|
| One gold run for a business date | run_gold gold/harness.py:127 | run_gold · scripts/run_hop.py:78gold_day · gold/daybyday.py:57 |
Silver is clean and complete, but it is still shaped like the sources: one table per source table, with every version of every row. Gold is shaped for questions. It holds dimensions, which describe things such as dealers, materials and presses, and facts, which hold the numbers people add up, such as order lines delivered on time or tyres scrapped per hour.
Nobody writes a job per gold table. One piece of code, the gold harness, builds all of them. A harness is a fixed frame that does the reading, the checking and the writing, and calls a small piece of table-specific logic in the middle. What changes from one table to the next is a manifest, which is a short file of settings, and for a fact one model, which is a single query or function.
Reader, processor and writer
The harness has the same three parts as the silver runner, and the code keeps them apart on purpose.
| Part | What it is in the code | What it may do |
|---|---|---|
| Reader | The read edge, GoldTables.silver and GoldTables.gold | Reads every row silver holds for an input, and the gold table as it is published now. Nothing else reads. |
| Processor | The fact's model, called through GoldEngine.build, and the engine operations around it (dimension shaping, key lookup, window numbers, checks) | Takes rows and gives rows or small numbers back. It never reads a table and never writes one. |
| Writer | The write edge, GoldTables.write_dimension and GoldTables.replace_windows | Replaces a whole dimension, or exactly the named windows of a fact, in one transaction. |
src/dataplatform/gold/ports.py:20The read and write edges as the harness sees them. An adapter for Spark and one for DuckDB fill them in.class GoldTables(Protocol):
"""The read and write edges of gold."""
def silver(self, dataset: str, columns: list[tuple[str, str]]) -> Frame:
"""Every row silver holds for `source.dataset` (every version, deleted ones marked). A table never written
(an input that has delivered nothing yet) is an empty frame of its contract `columns` (name, engine type)
and silver's own columns: an input with no rows is not an error."""
def silver_version(self, dataset: str) -> int | None:
"""The silver table's current version (commit or snapshot), recorded per gold run (INSP-01); None when
the table was never written or the format cannot say."""
def silver_at(self, dataset: str, version: int, columns: list[tuple[str, str]]) -> Frame:
"""The silver table as it was at `version` (time travel), to rebuild a past run's window (INSP-01)."""
def gold(self, plan: GoldPlan) -> Frame | None:
"""The gold table as published, or None before its first write."""
def write_dimension(self, plan: GoldPlan, rows: Frame) -> int:
"""Replace the whole dimension in one transaction (its keys are deterministic, so a rebuild changes nothing
that did not change). Returns the rows written."""
def replace_windows(self, plan: GoldPlan, rows: Frame, windows: list[str]) -> int:
"""Replace exactly these windows of a fact with `rows` in one transaction: rows of other windows are never
touched, rows of these windows not in `rows` are removed (ADR-006: replace, never append). Writing the same
rows twice changes nothing. Returns the rows written."""The order of one gold run
- Build the plans. Every enabled gold table of the domain is read from the metadata store and turned into a plan. A manifest the harness cannot run is refused here.
- Put dimensions first. The plans are sorted so that every dimension is built before any fact. A fact that names a dimension with no manifest is refused.
- Take a lease per table. Before a table is built the run takes a lease on it. A lease is a short lock with a time limit, kept in the metadata store. If another run holds it, this table is recorded as skipped and the run moves on.
- Build each dimension. Every silver version becomes one member with a stable integer key. The whole dimension is replaced. See Dimensions and Keys.
- Build each fact. The harness reads the inputs, calls the fact's model, looks up the dimension keys as of the event, decides which windows to rebuild and which are complete, runs the checks, and replaces only the windows that changed. See The harness, Facts and Windows and completeness.
- Record. Each step is one row in the run log. The table's bookmarks move only when its build succeeded.
src/dataplatform/gold/harness.py:127One gold run. The loop is the whole orchestration: plan, lease, build, release.def run_gold(store: MetadataStore, tables: GoldTables, engine: GoldEngine, domain: str, env: str, run_id: str,
as_of: str, tz: str, params: dict[str, str] | None = None, only: set[str] | None = None) -> GoldReport:
"""Build every gold table of the domain for business date `as_of` (or only the named tables)."""
ps = plans(store, domain)
ctx = Ctx(store, tables, engine, run_id, as_of, tz, {"business_tz": tz, **(params or {})}, {p.name: p for p in ps})
store.start_run(run_id, domain, env, "gold")
out = []
for p in ps:
if only is not None and p.name not in only:
continue
t0 = time.monotonic()
view = store.dataset_view(p.schema, p.name)
store.start_dataset_run(run_id, p.schema, p.name, view["fingerprint"], view)
lease = f"gold:{p.schema}.{p.name}"
holder = store.acquire_lease(lease, run_id, LEASE_SECONDS)
if holder is not None: # ORCH-04: another run writes this table now
one = GoldTable(p.name, "skipped", reason=busy(store, run_id, p.schema, p.name, lease, holder))
out.append(one)
continue
try:
one = _dimension(ctx, p) if p.kind == "gold_dimension" else _fact(ctx, p)
except (CheckFailed, GoldPlanError) as exc:
one = _fail(ctx, p, "gold.checks" if isinstance(exc, CheckFailed) else "gold.plan", exc)
except Exception as exc: # an engine or edge error: nothing published, the next run takes the window again
one = _fail(ctx, p, "gold.build", exc)
finally:
store.release_lease(lease, run_id)
one.seconds = round(time.monotonic() - t0, 1)
out.append(one)
return GoldReport(run_id, as_of, store.finish_run(run_id), out)Walk through it. plans returns the plans already sorted, dimensions first. For each one the run opens a dataset run in the run log and asks for the lease. A dimension goes to _dimension and a fact goes to _fact. A failed check or a refused plan is caught and recorded against the table, and the loop carries on with the next table. The lease is released in every case.
What is inside
| Page | What it explains |
|---|---|
| Manifests and models | What a gold manifest declares, how it becomes a plan, and what a model looks like. |
| The harness | The harness step by step: what it reads, the row ledger, the checks, the write and the run log. |
| Dimensions | How silver's type 2 history becomes dimension members with a validity span. |
| Keys | The stable integer keys, and why gold needs them on top of silver's hashes. |
| Facts | The three tyre facts, what a row of each means and how each is rebuilt. |
| Windows and completeness | Which windows a run rebuilds, when a window is complete, late data, restatements and withdrawn batches. |
The design decisions that shape this layer
| Decision | What it means | Why |
|---|---|---|
| One harness, one model per fact | The harness owns every read, key lookup, check and write. A person writes only the fact's own query or function. | The parts that are easy to get wrong, such as late data and reruns, are written and tested once. |
| Dimensions need no code | A dimension is one silver table reshaped by a common pipeline. Its manifest may not declare a model. | Every dimension gets keys, validity and reserved members in exactly the same way. |
| Facts are rebuilt by window and replaced, never appended | A fact names a window column. Each run recomputes a range of windows and replaces the ones whose content changed. | A rerun writes nothing, and a late or corrected row lands in the right place without special handling. |
| No fact row is ever dropped | A fact row whose dimension member has not arrived yet points at a reserved member with key -1 and is fixed when the member arrives. | Totals in gold always equal totals in the model's output. A missing dealer must not hide an order. |
| Checks come before the write | If a check fails, nothing is written for that table and the last good windows stay published. | Readers keep a correct older answer instead of getting a wrong newer one. |
| A window says whether it is complete | Every fact row carries _is_complete. A window is complete only when every input has loaded past its end. | A reader can tell a final number from one that is still filling. |
What a gold table looks like
The table below lists the columns the harness adds. The business columns of each gold table come from its contract.
| Table | Column | What it holds |
|---|---|---|
| Dimension | the key column, for example dealer_sk | The stable integer key of one version of one member. |
| Dimension | valid_from, valid_to, is_current | The time span in which this version was true. |
| Dimension | _source_valid_from, _is_deleted, _run_id | The real start of the version in the source, silver's delete mark, and the run that wrote the row. |
| Fact | one key column per dimension, for example dealer_sk | The member the row belongs to, as of the event. It is -1 when the member is unknown and -2 when the row has no business key. |
| Fact | _is_complete | True when every input of the row's window has arrived. |
| Fact | _run_id | The run that last changed the row's window. |
How we know it is right
Gold is compared with the oracle every day of a simulated run. The oracle is the simulated tyre company's own record of what is true, explained in Tests and proof. In the recorded run of 5 November 2026 (14 days at one tenth of full scale, local Spark), bronze, silver and gold equalled the oracle on every one of the 14 days. A full-scale attempt on 4 November 2026 equalled the oracle in every number through day 11 and then ran out of memory on the laptop in day 12's silver run, so the full-scale 14-day check is still owed. Both runs are written up in docs/W1D-GOLD.md.
- The full-scale 14-day gold run against the oracle has not completed. The recorded complete run is at one tenth of full scale.
- Gold has no settings in the settings registry. The lookback, the orphan limit and the completeness choice live only in the manifest.
- A manifest can say
materializeis something other than a table, and the plan checks that such a table has one input and no corrections, but nothing builds a materialised view. Every tyre table is a table. - The press readings stream feeds no gold table yet.