Data PlatformZoom level 2
Windows: what is rebuilt, what is complete, and what counts as a restatement
A window is one slice of a fact along its time column: one day of promised dates, one month of sales, one hour of curing. The harness thinks about a fact one window at a time. This page answers four questions about a window. Is it rebuilt in this run? Is it complete? Did its numbers change after readers were told they were final? What happens when a batch is withdrawn?
Without these rules, a dashboard would show a half-loaded day as if it were final, and a number that changed overnight would change without any record that it did.
Windows are named by text
The harness never handles the engine's date types. It names a window by a piece of text. A day is 2026-11-12. A month is its first day, 2026-10-01. An hour is its start in UTC, 2026-11-12T05:00:00Z. A day or a month starts at midnight in the business time zone. An hour is an instant and has no zone.
src/dataplatform/gold/windows.py:32The first instant after a window. Completeness is decided against this.def end(window: str, unit: str, tz: str) -> datetime:
"""The first instant after the window."""
if unit == "hour":
return start(window, unit, tz) + timedelta(hours=1)
d = date.fromisoformat(window[:10])
nxt = _month_add(d, 1) if unit == "month" else d + timedelta(days=1)
return datetime(nxt.year, nxt.month, nxt.day, tzinfo=ZoneInfo(tz))src/dataplatform/gold/windows.py:41The window an instant falls in.def of(t: datetime, unit: str, tz: str) -> str:
"""The window an instant falls in."""
if unit == "hour":
return t.astimezone(UTC).replace(minute=0, second=0, microsecond=0).strftime(HOUR_FORMAT)
local = t.astimezone(ZoneInfo(tz)).date()
return (local.replace(day=1) if unit == "month" else local).isoformat()Which windows a run rebuilds
src/dataplatform/gold/windows.py:49The first window of the range: the lookback counted back from the business date.def range_start(as_of: str, unit: str, lookback: int, tz: str) -> str:
"""The first window the recompute rebuilds on business date `as_of`: `lookback` units back (ADR-006)."""
d = date.fromisoformat(as_of)
if unit == "month":
return _month_add(d.replace(day=1), -lookback).isoformat()
if unit == "day":
return (d - timedelta(days=lookback)).isoformat()
day_end = datetime(d.year, d.month, d.day, tzinfo=ZoneInfo(tz)) + timedelta(days=1)
return of(day_end - timedelta(hours=lookback), "hour", tz)For days and months the lookback is counted back from the business date itself. For hours it is counted back from the end of the business date, so a 48-hour lookback covers the business date and the day before it. The table shows the three tyre facts for a run on 22 November 2026, with the business time zone of India. The values are the ones the test table checks.
| Fact | Unit and lookback | First window rebuilt |
|---|---|---|
fact_otif | day, 35 | 2026-10-18 |
fact_sales | month, 3 | 2026-08-01 |
fact_scrap | hour, 48 | 2026-11-20T18:00:00Z |
src/dataplatform/gold/harness.py:364The full range: from the first window on, plus named windows outside it.def _range(ctx: Ctx, p: GoldPlan, got: _Inputs, published: Frame | None) -> tuple[str, set[str]]:
"""The windows this run recomputes: the lookback from as_of, plus the windows active plans name and those with
rows on the unknown member; every window when a withdrawn batch's plan says so (VOID-01)."""
if p.window is None:
raise GoldPlanError(f"{p.name}: a fact needs a window")
first = W.range_start(ctx.as_of, p.window.unit, p.window.lookback, ctx.tz)
if EVERY_WINDOW in got.forced: # a withdrawn batch: recompute every window, replace only those that changed
first = BEFORE_ALL
return first, (got.forced - {EVERY_WINDOW}) | ctx.engine.windows_with_unknown(p, published)The range has no upper end. Every window from the first one on is recomputed, including future ones. On top of that, three kinds of window are added even when they are older than the lookback.
| Extra window | Why it is added |
|---|---|
| A window a correction plan names | A correction to an old order must reach its window however old it is. |
| A published window with a row on the unknown member | When the missing dealer arrives, the row must be given its real key. |
| Every window | When a batch was withdrawn, any window may have held its rows. |
Recomputing a window does not mean writing it. The write replaces only the windows whose content changed, as The harness explains.
Completeness: is every input in?
A window is complete when every input of the fact has loaded past the window's end. Until then its rows carry _is_complete = false. The harness does not look at the data to decide this. It compares the window's end with each input's watermark, which is the point in source time up to which silver has taken that input's deliveries.
src/dataplatform/gold/windows.py:65One input has caught up with a window when its watermark is at or after the window's end.def caught_up(window: str, unit: str, tz: str, watermark: datetime | None) -> bool:
"""One input has loaded past the window's end (ADR-018). No watermark: it has loaded nothing yet."""
return watermark is not None and end(window, unit, tz) <= watermarksrc/dataplatform/gold/harness.py:270A window is complete only if no input is still waited for.def _complete(ctx: Ctx, p: GoldPlan, window: str, got: _Inputs) -> tuple[bool, list[str]]:
"""ADR-018: every input caught up past the window's end (per-feed inputs: every expected feed delivered it).
A row the build says still waits for an input (`_input_complete` false) is flagged on its own (finish)."""
assert p.window is not None
unit, waits = p.window.unit, []
for ds, mark in got.marks.items():
if not W.caught_up(window, unit, ctx.tz, mark):
waits.append(ds)
for ds, (feeds, expected) in got.feeds.items():
owed = {m for m, first in expected.items() if first < W.end(window, unit, ctx.tz)} if expected else set(feeds)
ok, missing = W.feeds_caught_up(window, unit, ctx.tz, feeds, owed)
if not ok:
waits.append(f"{ds} ({len(missing)} feeds, e.g. {', '.join(missing[:3])})")
return not waits, waitsA worked example
Three orders have promised dates of 9, 10 and 12 November. Silver has taken the order deliveries for 9 and 10 November. The watermark is therefore the end of 10 November. Each row below is one window of the fact.
| Window | Window ends | Watermark at or after the end? | _is_complete |
|---|---|---|---|
| 2026-11-09 | end of 9 Nov | yes | true |
| 2026-11-10 | end of 10 Nov | yes | true |
| 2026-11-12 | end of 12 Nov | no | false |
Inputs that arrive one file per member
The dealer sell-out input is different. It is not one delivery a day. It is one file per dealer per month, and the dealers send them on different days. A single watermark cannot say whether October is in. So ingestion keeps one bookmark per dealer, holding the newest month that dealer has sent, and a month is complete when every dealer who owes a file has sent it.
src/dataplatform/gold/windows.py:70Complete when every expected feed has reached this window. The feeds still missing are returned.def feeds_caught_up(window: str, unit: str, tz: str, feeds: dict[str, datetime | None],
expected: set[str]) -> tuple[bool, list[str]]:
"""A per-feed input (one file per dealer): each expected feed's newest data time falls in this window or a
later one. Returns (complete, feeds still missing)."""
missing = sorted(f for f in expected if (t := feeds.get(f)) is None or of(t, unit, tz) < window)
return not missing, missingsrc/dataplatform/gold/harness.py:256Who owes a file: the members of the fact's dimension, each from its first version on.def _expected_feeds(ctx: Ctx, p: GoldPlan, feeds: dict[str, datetime | None]) -> dict[str, datetime]:
"""The members a per-feed input owes a file for: those of the fact's dimension the feed keys belong to, from
their first version in the source (a dealer that joined later owes nothing for earlier months). Without such a
dimension, the feeds seen so far."""
for d in p.dimensions:
dim = ctx.tables.gold(ctx.plans[d.name])
if dim is None or len(d.business_key) != 1:
continue
seen = ctx.engine.first_seen(dim, d.business_key[0])
if set(feeds) <= set(seen):
return seen
return {}The second function answers who owes a file. It looks for a dimension of the fact whose business keys include every feed name. For the sales fact that is the dealer dimension. Each dealer owes files from the start of its first version, so a dealer who joined in November owes nothing for October.
Rows that wait on their own
A model may return a column named _input_complete. A row where it is false is flagged incomplete even when its window is complete. The scrap model uses this for production rows whose inspection result has not been filed.
src/dataplatform/adapters/spark_gold.py:208The flag on each row: the window's flag, and the row's own flag when the model gave one. def finish(self, plan: GoldPlan, facts: DataFrame, complete: dict[str, bool], run_id: str) -> DataFrame:
flags = F.create_map(*[x for w, ok in sorted(complete.items()) for x in (F.lit(w), F.lit(ok))]) \
if complete else F.create_map().cast("map<string,boolean>")
done = F.coalesce(flags[self.window_text(plan)], F.lit(False))
if "_input_complete" in facts.columns: # a row still waiting for its own input is flagged on its own
done = done & F.coalesce(F.col("_input_complete"), F.lit(True))
rows = facts.withColumn("_is_complete", done).withColumn("_run_id", F.lit(run_id))
return rows.select(*[F.col(c).cast(t).alias(c) for c, t in plan.columns])Flag or hold
The manifest key completeness.on_incomplete decides what happens to an incomplete window. With flag it is published with the flag false, and a reader or a metric chooses whether to count it. With hold it is not written until it is complete. All three tyre facts use flag.
Late data and restatements
A restatement is a change to a number that readers were entitled to treat as final. Not every change is one. A day that fills up during the day is only filling. The rule is in one function.
src/dataplatform/gold/windows.py:91When a rebuild counts as a restatement.def is_restatement(before: Numbers | None, after: Numbers | None, window_end: datetime,
last_published: datetime | None, waited_for_feed: bool = False) -> bool:
"""A rebuild restates a window when numbers readers were entitled to treat as final change:
- the window was published complete and its numbers change (a correction, a late change), or
- the window, already over when it was published, was flagged incomplete because a member's feed had not
delivered (a dealer's file, LATE-01), and the rebuild that completes it changes its numbers.
A provisional number still waiting for the ordinary lag of an input (the next hour's lab file), a period still
open, or a window seen for the first time is not restated. A complete window whose rows all left it is."""
if before is None:
return False
ended = last_published is not None and window_end <= last_published
seen_final = before.complete or (ended and waited_for_feed)
if after is None: # every row left the window (a correction moved them): what readers saw is gone
return before.complete
return after.complete and not before.same_numbers(after) and seen_final| Before the run | After the run | Restatement? | Why |
|---|---|---|---|
| Not published | Anything | No | Nobody saw an earlier number. |
| Complete | Complete, different numbers | Yes | A final number changed. |
| Complete | No rows left | Yes | What readers saw is gone. |
| Incomplete, the period was over, a member's file was missing | Complete, different numbers | Yes | The period had ended, so readers took the number as the month's number. |
| Incomplete, waiting for an input's ordinary delay | Complete, different numbers | No | The number was provisional and said so. |
| Incomplete | Still incomplete | No | Still provisional. |
| Incomplete, the period was still open | Complete | No | An open period completing is normal. |
| Incomplete | Complete, same numbers | No | Only the flag moved. |
The late dealer file, followed through
In the simulated tyre company, the dealers' October sell-out files arrive over the first five days of the run, and one is late on purpose. In the recorded run of 5 November 2026 at one tenth of full scale, October was flagged incomplete on days 1 to 4. On day 5 the last file arrived, the month became complete, and it was replaced and logged as a restatement: 7,392 rows became 9,404 and 602,101 units sold out became 765,689. On day 6 a dealer resent a file with the same content. Nothing was replaced and nothing was logged. The table is in docs/W1D-GOLD.md.
src/dataplatform/gold/harness.py:453What is written to the run log about corrections, restatements and completeness.def _record(ctx: Ctx, p: GoldPlan, got: _Inputs, restated: list[Restatement], incomplete: list[str],
why: dict[str, list[str]], counts: dict[str, int]) -> None:
for b in got.applied:
_step(ctx, p, "detect.HIST-05", "warned", {"windows": len(b.windows), "corrections": got.corrections},
f"correction plan {b.plan_id} applied from {b.activate_on}: windows {', '.join(b.windows)} rebuilt "
f"({b.reason[:200]}; recorded by {b.recorded_by})")
plans = ", ".join(b.plan_id for b in got.applied)
for r in restated: # RSTM-01: one structured record per changed window, read by ops_restatements
reason = f"correction plan {plans}" if r.detector == "HIST-05" else "late data"
nums = {f"{k}.before": v for k, v in r.before.items()} | {f"{k}.after": v for k, v in r.after.items()}
_step(ctx, p, RESTATEMENT_STEP, "warned", nums, f"window={r.window}; reason={reason}; detector={r.detector}")
for det in ("LATE-01", "HIST-05"):
mine = [r for r in restated if r.detector == det]
if mine:
_step(ctx, p, f"detect.{det}", "warned", {"windows_restated": len(mine)},
"restated: " + "; ".join(f"{r.window} {r.before} -> {r.after}" for r in mine)[:3900])
waits = sorted({w for ws in why.values() for w in ws})
_step(ctx, p, "gold.completeness", "succeeded", {"windows_incomplete": len(incomplete)},
(f"flagged incomplete: {', '.join(incomplete[:12])}{' ...' if len(incomplete) > 12 else ''}; waiting for "
f"{'; '.join(waits)[:1500]}") if incomplete else "every window complete")
_step(ctx, p, "gold.write", "succeeded", counts)Each restated window gets one gold.restatement row with every number before and after. The operations layer reads those rows into a table of restatements with one row per window and measure. Operations covers that table.
Corrections
Sometimes a value in the source was simply wrong and someone corrects it after the fact, for example a promised date that was keyed in a day early. The platform does not edit silver. The correction is recorded in the metadata store, and the gold harness lays it over the silver rows in memory before the model runs.
Corrections come in a plan. A plan names the dataset, the date from which its corrections are active, the windows it expects to change, a reason, and who recorded it.
src/dataplatform/meta/port.py:120A plan and one correction, as the metadata store hands them to the harness.@dataclass(frozen=True)
class BackfillPlan:
"""A plan that activates corrections on a date and names the gold windows to rebuild (ADR-017, DPC-0082)."""
plan_id: str
source: str
dataset: str
windows: tuple[str, ...] # the windows the recompute rebuilds when the plan activates
activate_on: str # business date (ISO) from which its corrections are active
reason: str
recorded_by: str
@dataclass(frozen=True)
class Correction:
"""One corrected value (DPC-0082: entity, field, value, applies from and to, who, why, status). The entity is the
plan's dataset; who and why are the plan's. `key` holds the business key columns and their values."""
key: dict[str, Any]
field: str
value: str | None
was: str | None = None
applies_from: str | None = None # business dates; None = always
applies_to: str | None = None
plan_id: str = "" # filled by the store
status: str = "recorded" # recorded until the plan's date, then active (filled by the store)src/dataplatform/gold/harness.py:211Which corrections are applied to one input in this run.def _corrections(ctx: Ctx, p: GoldPlan, dataset: str, active: list[BackfillPlan]) -> tuple[list[Correction], int]:
"""Active corrections for one input. When a correction moves a row between windows (it corrects the fact's own
window column), it is applied only if its plan names both windows: a correction never restates a window by
surprise (ADR-017). Those it cannot apply are counted."""
source, name = dataset.split(".", 1)
named = {b.plan_id: set(b.windows) for b in active}
out, skipped = [], 0
for c in ctx.store.corrections(source, name, ctx.as_of):
if c.status != "active" or c.plan_id not in named:
continue
moves = p.window is not None and c.field == p.window.column
if moves and not {str(c.was), str(c.value)} <= named[c.plan_id]:
skipped += 1
continue
out.append(c)
return out, skippedThe rule in the middle is the important one. If a correction changes the fact's own window column, it moves a row from one window to another. It is applied only if the plan names both the old window and the new one. Otherwise it is skipped and counted in corrections_not_applied. A correction can therefore never change a window that the person who approved the plan did not expect to change.
src/dataplatform/adapters/spark_gold.py:145Laying corrections over silver rows: a small table of fixes joined on the business key. def overlay(self, frame: DataFrame, corrections: list[Correction], keys: tuple[str, ...]) -> DataFrame:
types = {f.name: f.dataType.simpleString() for f in frame.schema.fields} # not .dtypes: not on every engine
for fld in sorted({c.field for c in corrections}):
mine = [c for c in corrections if c.field == fld]
fields = {f.name: f.dataType for f in frame.schema.fields} # StructType[name] is not on every engine
schema = T.StructType([*[T.StructField(k, fields[k]) for k in keys],
T.StructField("_fix", T.StringType())])
fixes = self.spark.createDataFrame([(*[c.key[k] for k in keys], c.value) for c in mine], schema)
# shiftleft: allow SL-COST-CODE-14 -- joined on the business key columns `keys` (a USING join)
frame = (frame.join(fixes.hint("broadcast"), list(keys), "left")
.withColumn(fld, F.coalesce(F.col("_fix").cast(types[fld]), F.col(fld))).drop("_fix"))
return frameIn the recorded run, a planted correction to promised dates became active on day 11. The plan was applied on that day, the windows from 13 to 18 November were restated, and no window outside the plan changed.
A withdrawn batch
A batch that turns out to be bad after it was published can be withdrawn. Withdrawing is also called voiding. It marks the batch void, so no reader takes it again, and records a plan whose window list is a star, meaning every window.
src/dataplatform/recovery.py:18Withdrawing a batch: void it, and record a plan that names every window.def withdraw(store: MetadataStore, source: str, dataset: str, batch_key: str, *, reason: str, approved_by: str,
today: date) -> BackfillPlan:
store.void_batch(source, dataset, batch_key, reason, approved_by)
plan = BackfillPlan(f"void-{batch_key[:40]}", source, dataset, (EVERY_WINDOW,), today.isoformat(),
f"VOID-01: batch {batch_key} withdrawn: {reason}", approved_by)
store.record_corrections(plan, [])
return plan- Withdraw. The batch is voided with a reason and the name of who approved it.
- Rebuild silver. Silver for that dataset is rebuilt from bronze. The reader skips the void batch, so its rows are gone.
- Run gold as usual. Every fact that reads the dataset sees the plan, sets its range to every window, recomputes all of them, and replaces only the ones whose content changed.
- Read the log. Each changed window that had been complete is logged as a restatement whose reason names the plan.
No row is deleted by hand at any point. The withdrawn rows leave gold because the rebuild no longer produces them.
The settings that drive it
| Where | Key | What it means | Values in the tyre example |
|---|---|---|---|
| Fact manifest | window.column | The column a fact is sliced by. | promised_date, sale_month, cure_hour |
| Fact manifest | window.unit | The size of one window. | day, month, hour |
| Fact manifest | window.lookback | How many windows back each run recomputes. Default 0. | 35, 3, 48 |
| Fact manifest | completeness.on_incomplete | Publish an incomplete window flagged, or hold it back. Default flag. | flag |
| Fact manifest | corrections | Whether corrections are laid over the inputs. Default false. | true, true, false |
| Domain | business_timezone | Where a day and a month start. | Asia/Kolkata |
| Run | the business date | The date the run is for. The range is counted back from it. | one per day |
| Metadata store | silver's bookmark and the file log | The watermark of each input. | |
| Metadata store | per-feed bookmarks | The newest month each dealer has sent. |
The date dimension
Questions about windows usually need a calendar: which fiscal quarter a day is in, whether it was a working day. No source sends one, so the platform generates it. It is one row per day, keyed by the date and by a number such as 20261113. A fact can join it on any of its date columns without any change to the fact.
src/dataplatform/gold/calendar.py:26The fiscal year, quarter and month of a day, for a fiscal year that starts in a given month.def fiscal(d: date, start_month: int) -> tuple[int, int, int]:
"""(fiscal year, fiscal quarter, fiscal month) for a fiscal year starting on the 1st of start_month; the fiscal
year is named by the calendar year it ends in (India: April 2026 to March 2027 is fiscal year 2027)."""
if not 1 <= start_month <= 12: # noqa: PLR2004 -- months
raise ValueError(f"fiscal start month {start_month} is not 1 to 12")
shifted = (d.month - start_month) % 12
year = d.year + (1 if start_month > 1 and d.month >= start_month else 0)
return year, shifted // 3 + 1, shifted + 1The fiscal year is named by the calendar year it ends in. With a start in April, 1 April 2026 is the first day of fiscal year 2027, and 14 November 2026 is in fiscal quarter 3, fiscal month 8.
src/dataplatform/gold/calendar.py:36One row per day. Holidays are a list the caller passes in. Nothing is guessed.def calendar_rows(start: date, end: date, fiscal_start_month: int = 4,
holidays: Iterable[date] = (), weekend: tuple[int, ...] = (5, 6)) -> list[dict[str, Any]]:
"""One row per day from start to end inclusive. `weekend` holds weekday numbers (Monday = 0)."""
if end < start:
raise ValueError(f"calendar end {end} is before start {start}")
off = set(holidays)
rows = []
d = start
while d <= end:
iso_year, iso_week, _ = d.isocalendar()
fy, fq, fm = fiscal(d, fiscal_start_month)
is_weekend, is_holiday = d.weekday() in weekend, d in off
rows.append({
"date_key": d.year * 10000 + d.month * 100 + d.day, "date": d, "year": d.year,
"quarter": (d.month - 1) // 3 + 1, "month": d.month, "month_name": MONTHS[d.month - 1],
"year_month": f"{d.year}-{d.month:02d}", "iso_week": iso_week, "iso_year": iso_year,
"day_of_month": d.day, "day_of_week": d.weekday() + 1, "day_name": DAYS[d.weekday()],
"is_weekend": is_weekend, "is_holiday": is_holiday, "is_working_day": not (is_weekend or is_holiday),
"fiscal_year": fy, "fiscal_quarter": fq, "fiscal_month": fm,
})
d += timedelta(days=1)
return rowsThe run script writes the calendar whole on every run, for five years either side of the simulated company's start date, with the fiscal year starting in April.
How it is tested
The window arithmetic, completeness and the restatement rule are pure functions, so tests/dataplatform/test_gold_plan.py tests them as tables with no engine. tests/dataplatform/test_gold_harness.py then runs the late file, the correction and the withdrawn batch through the real harness. tests/dataplatform/test_gold_calendar.py covers the calendar, and tests/dataplatform/test_ops_restatements.py covers the table of restatements.
tests/dataplatform/test_gold_plan.py:203The restatement rule as a table: numbers before, numbers after, when it was last published, whether a feed was waited for, and the verdict. ("first publication", None, N(5, (("q", 1.0),), True), AFTER, False, False),
("same numbers", N(5, (("q", 1.0),), True), N(5, (("q", 1.0),), True), AFTER, False, False),
("a complete window changes (a correction)", N(5, (("q", 1.0),), True), N(5, (("q", 2.0),), True), AFTER,
False, True),
("a late member's file completes an ended month (LATE-01)", N(5, (("q", 1.0),), False),
N(6, (("q", 2.0),), True), AFTER, True, True),
("an input's ordinary lag completes an ended hour", N(5, (("q", 1.0),), False), N(5, (("q", 2.0),), True),
AFTER, False, False),
("a provisional number of an ended month changes", N(5, (("q", 1.0),), False), N(6, (("q", 2.0),), False),
AFTER, True, False),
("an open period completes", N(5, (("q", 1.0),), False), N(6, (("q", 2.0),), True), BEFORE, True, False),
("only the flag moves", N(5, (("q", 1.0),), False), N(5, (("q", 1.0),), True), AFTER, True, False),
("every row moved out of a complete window", N(5, (("q", 1.0),), True), None, AFTER, False, True),
("every row moved out of a provisional window", N(5, (("q", 1.0),), False), None, AFTER, False, False),
],
)
def test_is_restatement(case, before, after, published, feed, restated):
assert W.is_restatement(before, after, END, published, feed) is restatedtests/dataplatform/test_gold_harness.py:274Dealer D-1 sends October on day 1 and D-2 on day 5. The month is incomplete, then complete and restated from 5 units to 12. A resend changes nothing.def test_a_late_member_file_completes_and_restates_the_month(monthly):
monthly.silver("feed", FEED, [{"dealer_code": "D-1", "month": OCT, "units": 5}])
feed(monthly.store, DAYS[0], ["D-1"])
day1 = monthly.run(DAYS[0], only={"dim_dealer", "fact_monthly"}).by_name()["fact_monthly"]
assert day1.incomplete == ["2026-10-01"] and not day1.restated
assert "src.feed (1 feeds, e.g. D-2)" in next(s["message"] for s in monthly.steps("gold-1")
if s["step"] == "gold.completeness")
monthly.silver("feed", FEED, [{"dealer_code": "D-1", "month": OCT, "units": 5},
{"dealer_code": "D-2", "month": OCT, "units": 7}])
feed(monthly.store, DAYS[4], ["D-2"])
day5 = monthly.run(DAYS[4], only={"dim_dealer", "fact_monthly"}).by_name()["fact_monthly"]
assert day5.incomplete == []
assert [(r.window, r.detector) for r in day5.restated] == [("2026-10-01", "LATE-01")]
assert any(s["step"] == "detect.LATE-01" for s in monthly.steps("gold-2"))
[rst] = [s for s in monthly.steps("gold-2") if s["step"] == "gold.restatement"] # RSTM-01
assert "window=2026-10-01; reason=late data" in rst["message"]
assert (rst["counts"]["units.before"], rst["counts"]["units.after"]) == (5, 12)
resend = monthly.run(DAYS[5], only={"dim_dealer", "fact_monthly"}).by_name()["fact_monthly"]
assert (resend.replaced, resend.restated) == ([], [])tests/dataplatform/test_ops_restatements.py:18From run log rows to the table of restatements: one row per window and measure, with the change.@pytest.mark.parametrize(("case", "steps", "want"), [
("late data moved a published day",
[step("gold.restatement", {"rows.before": 10, "rows.after": 12, "qty.before": 100.0, "qty.after": 130.5},
"window=2026-11-10; reason=late data; detector=LATE-01")],
[("2026-11-10", "late data", "qty", 100.0, 130.5, 30.5), ("2026-11-10", "late data", "rows", 10.0, 12.0, 2.0)]),
("a correction plan", [step("gold.restatement", {"rows.before": 5, "rows.after": 5},
"window=2026-11-02; reason=correction plan P-7; detector=HIST-05")],
[("2026-11-02", "correction plan P-7", "rows", 5.0, 5.0, 0.0)]),
("other steps are not restatements", [step("gold.write", {"rows": 3}, "")], []),
])- Completeness only knows deliveries by business day. An input that silver loads several times a day still counts as loaded only through the end of the newest whole day it has taken.
- Whether an input arrives one file per member is inferred from ingestion's bookmarks. No manifest key declares it.
- A day on which each dealer's file is due is not known to the platform. October is flagged until the last dealer has sent, with no notion of a file being overdue.
- The specification says restatements are logged but not when a rebuild is one. The rule on this page is a decision taken in the code.
- A restatement caused by a withdrawn batch is recorded under the same detector name as a correction plan. Only the reason text tells them apart.
- Restatements live in the run log. There is no separate restatement table in the metadata store.
- The date dimension has no holidays for the tyre company. The run script passes none.
- Plant time zones are not declared anywhere the platform reads, so an hour window is always a UTC hour.