Data PlatformZoom level 2
The gold harness, step by step
The harness is the code that builds a gold table from its plan. For a fact it does everything except the business logic: it reads the inputs, calls the model, attaches dimension keys, decides which windows to rebuild, checks the result, writes it, and records what happened.
It exists so that the hard parts are written once. Without it, every fact would have its own handling of reruns, late rows and missing dimension members, and each would be wrong in its own way.
One fact build, in the order the code runs
- Read the table's bookmarks. The fact's own bookmarks say how far it has published and which correction plans it has already applied.
- Read each input. For every input the harness notes the silver table's version, reads every row silver holds, lays active corrections over the rows if the fact allows corrections, and works out how far that input has loaded in source time.
- Record what was read. Two rows go to the run log: the version of each input, and the text of the model with its parameter values. A past window can be rebuilt later from these two rows alone.
- Call the model. The model returns one row per grain with business keys and measures.
- Look up the keys. Each dimension's key is attached by a left join as of the event time. The row count before and after must be equal. The row ledger is written.
- Choose the windows. The range is the lookback from the business date, plus windows a correction plan names, plus published windows that still hold rows on the unknown member.
- Decide completeness. Each window in the range is complete only if every input has loaded past the window's end.
- Finish the rows. The harness adds
_is_completeand_run_id, and puts the columns in contract order. - Check. The grain must be unique, the share of rows on the unknown member must be under the limit, and the totals must equal the model's totals. A failure stops here and nothing is written.
- Replace what changed. A digest of each window's content is compared with the published window. Only windows whose digest differs are replaced, in one transaction.
- Log restatements. A changed window that readers had been entitled to treat as final is logged as a restatement, with its numbers before and after.
- Move the bookmarks. The input watermarks, how far gold has published, and the plans applied are saved together with the success status.
What it reads from silver, and how
The read edge hands over every row silver holds for an input: every version of every key, with deleted rows marked. It does not filter to the current rows and it does not read only what changed since last time. The model decides which rows it wants. An input that silver has never written comes back as an empty table with the right columns, so a fact can be built before all of its inputs have delivered.
src/dataplatform/gold/harness.py:229Reading the inputs of one fact: version, rows, corrections, and how far each input has loaded.def _inputs(ctx: Ctx, p: GoldPlan, done: dict[str, str]) -> _Inputs:
got = _Inputs({}, {}, {}, [], set())
for i in p.inputs:
got.versions[i.dataset] = ctx.tables.silver_version(i.dataset)
frame = ctx.tables.silver(i.dataset, _columns(ctx, i.dataset))
active = _active_plans(ctx, i.dataset)
withdrawn = [b for b in active if EVERY_WINDOW in b.windows] # VOID-01: every fact on the input reruns
got.applied += [b for b in withdrawn if f"plan.{b.plan_id}" not in done]
got.forced |= {EVERY_WINDOW for b in withdrawn if f"plan.{b.plan_id}" not in done}
if p.corrections:
active = [b for b in active if b not in withdrawn]
got.applied += [b for b in active if f"plan.{b.plan_id}" not in done]
got.forced |= {w for b in active for w in b.windows}
fixes, skipped = _corrections(ctx, p, i.dataset, active)
got.corrections, got.skipped_corrections = got.corrections + len(fixes), got.skipped_corrections + skipped
if fixes:
frame = ctx.engine.overlay(frame, fixes, tuple(ctx.store.business_keys(i.source, i.name)))
got.frames[f"{i.source}__{i.name}"] = frame
feeds = feed_marks(ctx.store, i.dataset)
if feeds:
got.feeds[i.dataset] = (feeds, _expected_feeds(ctx, p, feeds))
else:
got.marks[i.dataset] = silver_watermark(ctx.store, i.dataset, ctx.tz)
_trace(ctx, p, got)
return gotWalk through the loop. silver_version records the table version that was read. silver reads the rows. The lines about plans deal with corrections and withdrawn batches, which Windows and completeness explains. The frame is stored under the name the model will see, the source and dataset joined by two underscores. The last lines pick one of two ways to say how far the input has loaded. If ingestion keeps one bookmark per feed, for example one per dealer file, those are used. Otherwise the input has one watermark.
src/dataplatform/gold/harness.py:102How far an input has loaded: the end of the newest business day whose delivery silver has taken.def silver_watermark(store: MetadataStore, dataset: str, tz: str) -> datetime | None:
"""How far an input has loaded in source time (ADR-018): the end of the newest delivery window silver has taken
(silver's bookmark counts the file records it took; an empty delivery still moves it)."""
source, name = dataset.split(".", 1)
taken = int(store.bookmarks(source, name).get(SILVER_BOOKMARK, "0"))
days = [f.window for f in store.dataset_files(source, name)[:taken] if f.status == "loaded" and f.window]
return W.day_end(max(days), tz) if days else NoneA watermark here is the latest point in source time up to which an input is known to be loaded. It is not read from the data. It comes from the metadata store: silver's bookmark says how many file log records it has taken, the harness looks at the delivery window of each loaded record among those, and takes the end of the newest day. An empty delivery still moves it, which is right, because a day with no orders is still a day that has been loaded.
Where the as-of join on version validity happens
Silver keeps type 2 history: each version of a dealer has a span during which it was true. Gold uses that span in one place, the key lookup. The dimension build copies silver's span into valid_from and valid_to on each member. The key lookup then joins each fact row to the member whose span contains the fact's event time. This is called an as-of join: the row gets the dealer as the dealer was at that moment.
src/dataplatform/adapters/spark_gold.py:191The key lookup on Spark: one left join per dimension, on the business key and the validity span. def add_keys(self, plan: GoldPlan, facts: DataFrame, dimensions: dict[str, DataFrame]) -> DataFrame:
out = facts
for i, d in enumerate(plan.dimensions):
dim = dimensions[d.name].where(F.col(d.key) >= 0)
names = {c: f"_d{i}_{c}" for c in (d.key, *d.business_key, "valid_from", "valid_to")}
dim = dim.select(*[F.col(c).alias(a) for c, a in names.items()])
t = F.col(d.as_of).cast("timestamp")
on = [F.col(b) == F.col(names[b]) for b in d.business_key]
on += [F.col(names["valid_from"]) <= t, t < F.col(names["valid_to"])]
no_key = reduce(lambda a, b: a & b, [F.col(b).isNull() for b in d.business_key])
# shiftleft: allow SL-COST-CODE-14 -- `on` is the business key equality plus the validity range (ADR-005)
out = (out.join(dim, reduce(lambda a, b: a & b, on), "left")
.withColumn(d.key, F.when(no_key, F.lit(NOT_APPLICABLE))
.otherwise(F.coalesce(F.col(names[d.key]), F.lit(UNKNOWN))).cast("long"))
.drop(*names.values()))
return outThe lines that matter, in order. Reserved members are taken out of the dimension first, so a fact can never match them by accident. The dimension's columns are renamed with a prefix so they cannot clash with the fact's. The join condition is the business key being equal, plus valid_from <= t and t < valid_to. The span is closed at the start and open at the end, so an event exactly at a change belongs to the new version and can never match two. The join is a left join, so a fact row with no match is kept. The last expression sets the key: -2 if the fact has no business key at all, the member's key if one matched, and -1 if none did.
The row ledger
A join is the easiest place to lose or multiply rows without noticing. The harness therefore counts. It counts the rows read from each input, the rows the model returned, the rows after the key lookup, and for each dimension the rows that matched a member, the rows on the unknown member and the rows marked not applicable. These counts are the row ledger, and they are written to the run log as the step gold.ledger.
src/dataplatform/gold/harness.py:375Rows in must equal rows out, or the table is not published.def _key_lookup(ctx: Ctx, p: GoldPlan, got: _Inputs, built: Frame, dims: dict[str, Frame]) -> Frame:
"""Surrogate keys by the as-of joins (ADR-005), the rows-in = rows-out check and the row ledger (LEDG-01)."""
keyed = ctx.engine.hold(ctx.engine.add_keys(p, built, dims))
rows_in, rows_out = ctx.engine.count(built), ctx.engine.count(keyed)
culprit = "" if rows_in == rows_out else _which_join(ctx, p, built, dims, rows_in)
_step(ctx, p, "gold.keys", "succeeded" if rows_in == rows_out else "failed",
{"rows_in": rows_in, "rows_out": rows_out}, culprit)
_ledger(ctx, p, got, (rows_in, rows_out), keyed)
if rows_in != rows_out: # a fact row lost or multiplied by a join: never published
raise CheckFailed(f"{p.name}: {rows_in} rows built, {rows_out} after the key lookup; {culprit}")
return keyedsrc/dataplatform/gold/harness.py:399The ledger. Each join's three counts must add up to the rows after the lookup.def _ledger(ctx: Ctx, p: GoldPlan, got: _Inputs, sizes: tuple[int, int], keyed: Frame) -> dict[str, int]:
"""LEDG-01: rows read per input, rows the build gave, and per dimension join rows matched, on the unknown member
and not applicable. Each join's three counts must add up to the rows after the lookup, else the run fails."""
rows_in, rows_out = sizes
counts = {f"read.{ds}": ctx.engine.count(f) for ds, f in sorted(got.frames.items()) if f is not None}
counts.update(built=rows_in, after_keys=rows_out)
off = []
for key, t in ctx.engine.key_tally(p, keyed).items():
counts.update({f"{key}.{w}": n for w, n in t.items()})
if sum(t.values()) != rows_out:
off.append(f"{key}: {sum(t.values())} of {rows_out}")
balanced = not off and rows_in == rows_out
_step(ctx, p, LEDGER_STEP, "succeeded" if balanced else "failed", counts,
"" if balanced else f"the ledger does not balance: {'; '.join(off) or f'{rows_in} built, {rows_out} keyed'}")
if off:
raise CheckFailed(f"{p.name}: the ledger does not balance: {'; '.join(off)}")
return countsTwo rules make the ledger a check and not only a report. The rows built must equal the rows keyed. And for every dimension, matched plus unknown plus not applicable must equal the rows keyed. If the total moved, _which_join reruns the lookup one dimension at a time and names the first join that changed the count, so the message says which dimension has overlapping versions.
A worked ledger
Four order lines go in. One is for dealer D-9, who has no record yet. One has no dealer code. The left table is the model's output. The right table is the ledger the run log receives.
| order_id | dealer_code | dealer_sk after the lookup |
|---|---|---|
| O0 | D-1 | the key of D-1's version |
| O1 | D-9 | -1 |
| O2 | (none) | -2 |
| O3 | D-2 | the key of D-2's version |
| Ledger count | Value |
|---|---|
built | 4 |
after_keys | 4 |
dealer_sk.matched | 2 |
dealer_sk.unknown | 1 |
dealer_sk.not_applicable | 1 |
The checks before publishing
src/dataplatform/gold/harness.py:418Three checks. Any failure raises, and the caller writes nothing.def _checks(ctx: Ctx, p: GoldPlan, rows: Frame, built: Frame, first: str, also: set[str]) -> dict[str, int]:
"""Checks before publish (ADR-006): grain unique (KEY-01), orphan rate (LATE-02), reconcile with what the build
gave (no row or quantity lost by the harness). A failure raises: nothing is written."""
n = ctx.engine.count(rows)
counts = {"rows": n}
if p.grain_unique:
dup = ctx.engine.duplicates(rows, p.grain)
_step(ctx, p, "detect.KEY-01", "failed" if dup else "succeeded", {"rows": n, "duplicate_grain": dup},
f"{dup} rows repeat the grain {list(p.grain)}" if dup else "")
if dup:
raise CheckFailed(f"{p.name}: {dup} rows repeat the grain")
unknown = ctx.engine.unknown_rows(p, rows)
orphans = max(unknown.values(), default=0)
counts["orphan_rows"] = orphans
if orphans:
rate = orphans / n if n else 0.0
over = p.orphan_rate_max is not None and rate > p.orphan_rate_max
_step(ctx, p, "detect.LATE-02", "failed" if over else "warned",
{"orphan_rows": orphans, "rows": n, **{f"unknown_{k}": v for k, v in unknown.items()}},
f"{orphans} rows on the unknown member ({rate:.4%}; limit {p.orphan_rate_max}); re-keyed when the "
f"member arrives")
if over:
raise CheckFailed(f"{p.name}: orphan rate {rate:.4%} above {p.orphan_rate_max}")
if p.reconcile:
cols = tuple(c for _, c in p.reconcile.measures)
want = ctx.engine.totals(ctx.engine.only_windows(p, built, first, also), cols)
have = ctx.engine.totals(rows, cols)
bad = {c: (want[c], have[c]) for c in cols if abs(want[c] - have[c]) > p.reconcile.tolerance.get(c, 0.0)}
_step(ctx, p, "gold.reconcile", "failed" if bad else "succeeded",
{f"{c}_total": int(round(have[c])) for c in cols}, f"differs from the build: {bad}" if bad else "")
if bad:
raise CheckFailed(f"{p.name}: totals differ from the build: {bad}")
return counts| Check | Run log step | What fails it |
|---|---|---|
| Grain unique | detect.KEY-01 | Two rows share the grain columns. |
| Orphan rate | detect.LATE-02 | Rows on the unknown member are a warning. They fail the build only when their share is above orphan_rate_max. |
| Reconcile | gold.reconcile | A measure's total in the rows about to be written differs from its total in the model's output for the same windows by more than the tolerance. |
The reconcile check compares the harness with the model, not gold with silver. It proves that the key lookup, the window selection and the column ordering lost no quantity on the way.
The write: replace changed windows
src/dataplatform/gold/harness.py:298The middle of _fact: the range, completeness, the checks, and the write. published = ctx.tables.gold(p)
first, also = _range(ctx, p, got, published)
candidate = ctx.engine.window_numbers(p, keyed, first, also)
waiting = ctx.engine.rows_incomplete(p, keyed)
complete, why = {}, {}
for w in candidate:
complete[w], why[w] = _complete(ctx, p, w, got)
if w in waiting:
why[w].append("rows waiting for an input")
rows = ctx.engine.hold(ctx.engine.finish(p, ctx.engine.only_windows(p, keyed, first, also), complete, ctx.run_id))
try:
counts = _checks(ctx, p, rows, built, first, also)
except CheckFailed:
ctx.engine.release(built, keyed, rows)
raise
before = ctx.engine.window_numbers(p, published, first, also) if published is not None else {}
after = ctx.engine.window_numbers(p, rows, first, also)
changed = sorted(w for w in set(before) | set(after) if before.get(w, (None, ""))[1] != after.get(w, (None, ""))[1])
if p.on_incomplete == "hold":
changed = [w for w in changed if complete.get(w, True) and w not in waiting]
written = ctx.tables.replace_windows(p, ctx.engine.only_windows(p, rows, AFTER_ALL, set(changed)), changed) \
if changed else 0
ctx.engine.release(built, keyed, rows)The decision to write is made from digests. window_numbers returns, for each window, its row count, the sums of its measures and a digest, which is one number computed from every column of every row except _run_id. The harness computes this for the published table and for the new rows. A window is in changed only if the two digests differ. If the fact's manifest says hold, incomplete windows are taken out of the list. Then one call replaces exactly those windows.
src/dataplatform/adapters/spark_gold.py:360The write edge on Delta: one overwrite limited to the named windows. def replace_windows(self, plan: GoldPlan, rows: DataFrame, windows: list[str]) -> int:
assert plan.window is not None
bad = [w for w in windows if not _WINDOW.match(w)]
if bad:
raise ValueError(f"not window names: {bad[:3]}")
path = self.path(plan)
if not (path / "_delta_log").exists():
rows.write.format("delta").mode("overwrite").save(str(path))
return int(self.spark.read.format("delta").load(str(path)).count())
col = _safe(plan.window.column)
if plan.window.unit == "hour":
values = ", ".join(f"TIMESTAMP'{w[:10]} {w[11:19]}'" for w in windows)
else:
values = ", ".join(f"DATE'{w}'" for w in windows)
(rows.write.format("delta").mode("overwrite").option("replaceWhere", f"`{col}` IN ({values})")
.save(str(path)))
return int(rows.count())The write is an overwrite with a condition on the window column, so rows of the named windows are removed and the new rows put in their place in one commit. Rows of other windows are not touched. A window that is named but has no new rows becomes empty, which is how an order that was deleted at the source leaves gold. The window names are checked against a strict pattern before they are placed in the condition, so no data value can ever become part of the statement.
The digest leaves out _run_id. A second run over the same silver produces the same digest for every window, so changed is empty and nothing is written. A side effect is useful: the _run_id on a row tells you which run last changed its window.
What goes into the run log
Each row below is one step the harness records for a table, in the order they are written. The step names are the ones in the code.
| Step | Written when | What it carries |
|---|---|---|
gold.inputs | After the inputs are read | The silver version of each input, and the business date of the run. |
gold.query | After the inputs are read | The model's SQL with the parameter values shown, or the name of the dataframe function. |
gold.keys | After the key lookup | Rows in and rows out. Failed if they differ, with the join that caused it. |
gold.ledger | After the key lookup | Rows read per input, rows built, rows keyed, and three counts per dimension. |
detect.KEY-01 | In the checks, and once per dimension | Duplicate grain rows, or duplicate dimension keys. |
detect.LATE-02 | In the checks, only when there are orphans | Rows on the unknown member, per dimension key. |
gold.reconcile | In the checks | The totals of the reconcile measures. |
detect.HIST-05 | When a correction plan is applied for the first time | The plan, its windows, its reason and who recorded it. |
gold.restatement | Once per restated window | Each number before and after, with the window and the reason. |
detect.LATE-01 | When late data restated a window | The windows restated. |
gold.completeness | Always | How many windows are incomplete and what each waits for. |
gold.write | Last | All the counts of the build. |
src/dataplatform/gold/harness.py:321The end of _fact: restatements, the counts, and the bookmarks saved with the success status. last = _time(done.get("published_through", ""))
waited = set(filter(None, done.get("feeds_waiting", "").split(",")))
applied_windows = {w for b in got.applied for w in b.windows}
if EVERY_WINDOW in applied_windows:
applied_windows |= set(changed)
restated = []
for w in changed:
b, a = before.get(w, (None, ""))[0], after.get(w, (None, ""))[0]
if W.is_restatement(b, a, W.end(w, unit, tz), last, w in waited):
restated.append(Restatement(p.name, w, "HIST-05" if w in applied_windows else "LATE-01",
_numbers(b), _numbers(a)))
incomplete = sorted(w for w, ok in complete.items() if not ok or w in waiting)
counts.update(windows_rebuilt=len(after), windows_replaced=len(changed), rows_written=written,
windows_incomplete=len(incomplete), restatements=len(restated), corrections=got.corrections,
corrections_not_applied=got.skipped_corrections)
_record(ctx, p, got, restated, incomplete, why, counts)
marks = {f"input.{ds}": m.isoformat() for ds, m in got.marks.items() if m is not None}
marks |= {f"input.{ds}": min(t for t in f.values() if t is not None).isoformat() for ds, (f, _) in got.feeds.items()}
marks |= {f"plan.{b.plan_id}": ctx.as_of for b in got.applied}
marks["published_through"] = W.day_end(ctx.as_of, tz).isoformat()
marks["feeds_waiting"] = ",".join(sorted(w for w in incomplete if any(" feeds, " in x for x in why[w])))
ctx.store.finish_dataset_run(ctx.run_id, p.schema, p.name, "succeeded", counts, bookmarks=marks)
return GoldTable(p.name, "succeeded", counts, changed, incomplete, restated, [b.plan_id for b in got.applied])The bookmarks of a fact are saved in the same call that marks the build succeeded. The table lists them.
| Bookmark | What it holds |
|---|---|
input.<dataset> | The watermark of each input in source time, as this run saw it. |
published_through | The end of the business date this run was for. |
feeds_waiting | The windows that are incomplete because a member's file has not arrived. |
plan.<id> | The business date on which a correction plan was first applied, so it is logged once. |
When something fails
src/dataplatform/gold/harness.py:171A failure is recorded against the table and the run carries on with the next one.def _fail(ctx: Ctx, p: GoldPlan, step: str, exc: Exception) -> GoldTable:
msg = f"{type(exc).__name__}: {str(exc)[:1500]}"
ctx.store.log_error(ctx.run_id, p.schema, p.name, step, type(exc).__name__, msg)
ctx.store.finish_dataset_run(ctx.run_id, p.schema, p.name, "failed", message=msg)
return GoldTable(p.name, "failed", reason=msg)A failed check is recorded under the step gold.checks, a refused plan under gold.plan, and any other error under gold.build. In all three cases nothing was written and no bookmark moved, so the next run tries the same windows again.
Rebuilding a past window from the run log
Because every fact run records its input versions and its query, src/dataplatform/gold/inspect.py can rebuild one window of a past run. It reads the two run log rows, reads each silver input as it was at that version, runs the same model and key lookup, and returns the window's numbers beside what gold publishes now. It writes nothing.
src/dataplatform/gold/harness.py:354The two rows that make a past run reproducible.def _trace(ctx: Ctx, p: GoldPlan, got: _Inputs) -> None:
"""INSP-01: the exact inputs (silver table and version) and the query this run built from, so one window can
be rebuilt later from the run log alone (gold/inspect.py). An unknown version is recorded as -1."""
versions = {f"version.{ds}": (-1 if v is None else v) for ds, v in sorted(got.versions.items())}
named = ", ".join(f"{ds}@{v}" for ds, v in sorted(got.versions.items()))
_step(ctx, p, INPUTS_STEP, "succeeded", versions, f"as_of={ctx.as_of}; inputs={named}")
_step(ctx, p, QUERY_STEP, "succeeded", {}, ctx.engine.query_text(p, ctx.params))How it is tested
tests/dataplatform/test_gold_harness.py runs the real harness on local Spark with a metadata store held in memory and a small made-up shop: dealers, orders by day, and a monthly file per dealer. tests/dataplatform/test_gold_ports.py is the contract table for the read and write edges, which every adapter must pass.
tests/dataplatform/test_gold_harness.py:95Three orders on three days. A rerun writes nothing. A delete and a change replace only their own two windows.def test_replace_never_append(shop):
shop.silver("order", ORDER, [order("A", "D-1", 9), order("B", "D-1", 10), order("C", "D-2", 11)])
silver_loaded(shop.store, "src.order", DAYS[:3])
first = shop.run(DAYS[2]).by_name()["fact_orders"]
before = shop.facts()
again = shop.run(DAYS[2]).by_name()["fact_orders"]
assert (first.counts["windows_replaced"], again.counts["windows_replaced"]) == (3, 0)
assert shop.facts() == before # the rerun wrote nothing: not even the run id moved
# B is deleted and C changes: their windows are replaced whole; A's window keeps the first run's rows
shop.silver("order", ORDER, [order("A", "D-1", 9), {**order("B", "D-1", 10), "_is_deleted": True},
order("C", "D-2", 11, qty=5)])
third = shop.run(DAYS[2]).by_name()["fact_orders"]
after = {r["order_id"]: r for r in shop.facts()}
assert sorted(third.replaced) == ["2026-11-10", "2026-11-11"]
assert set(after) == {"A", "C"} and after["C"]["qty"] == 5
assert after["A"]["_run_id"] == "gold-1" and after["C"]["_run_id"] == "gold-3"The last two lines are the sharpest. Order A's window was not replaced, so its row still carries the first run's id. Order C's window was replaced, so its row carries the third run's id.
tests/dataplatform/test_gold_harness.py:214Two dealer versions that overlap would multiply a fact row. The key step fails and gold is unchanged.def test_a_failed_check_holds_the_last_good_windows(shop):
shop.silver("order", ORDER, [order("A", "D-1", 9)])
silver_loaded(shop.store, "src.order", DAYS[:1])
shop.run(DAYS[0])
good = shop.facts()
# two dealer versions that overlap would fan a fact row out: the key step fails and nothing is written
shop.silver("dealer", DEALER, [*DEALERS, {"dealer_code": "D-1", "region": "east", "_valid_from": ts(1),
"_row_key": "dup"}])
shop.silver("order", ORDER, [order("A", "D-1", 9, qty=9)])
t = shop.run(DAYS[0]).by_name()["fact_orders"]
assert t.status == "failed" and "after the key lookup" in t.reason
assert "the join to dim_dealer" in t.reason # LEDG-01: the join that multiplied the rows is named
assert shop.facts() == good- Every run reads every silver row of every input and runs the model over all of it. Only the write is limited to changed windows. The manifest's comment about finding only the changed days describes an intent the code does not have yet.
- The inspect tool reads dimensions as they are now and does not replay corrections, so a window that a correction plan rebuilt shows its uncorrected numbers there.
- The window digest is computed with a hash function of the engine, so digests can be compared between two runs of one engine but not between Spark and DuckDB.
- The tolerance of the reconcile check is looked up by column name, while the tyre manifests write it by measure name. For those manifests the tolerance is 0 either way, so nothing shows today.