cleandataengineering.aiData Platform, from the big picture to the code
You are here: Start here › Silver to gold › Windows and completeness

Data PlatformZoom level 2

Windows: what is rebuilt, what is complete, and what counts as a restatement

A window is one slice of a fact along its time column: one day of promised dates, one month of sales, one hour of curing. The harness thinks about a fact one window at a time. This page answers four questions about a window. Is it rebuilt in this run? Is it complete? Did its numbers change after readers were told they were final? What happens when a batch is withdrawn?

Without these rules, a dashboard would show a half-loaded day as if it were final, and a number that changed overnight would change without any record that it did.

18 Oct...20 Nov21 Nov22 Nov23 Nov24 Novfirst window in rangebusiness date of the runRebuilt and can be complete: inputs have loaded past these daysRebuilt, flagged incomplete
A run for 22 November starts 35 days back. Windows after the business date are built too, but they cannot be complete yet.

Windows are named by text

The harness never handles the engine's date types. It names a window by a piece of text. A day is 2026-11-12. A month is its first day, 2026-10-01. An hour is its start in UTC, 2026-11-12T05:00:00Z. A day or a month starts at midnight in the business time zone. An hour is an instant and has no zone.

src/dataplatform/gold/windows.py:32The first instant after a window. Completeness is decided against this.
def end(window: str, unit: str, tz: str) -> datetime:
    """The first instant after the window."""
    if unit == "hour":
        return start(window, unit, tz) + timedelta(hours=1)
    d = date.fromisoformat(window[:10])
    nxt = _month_add(d, 1) if unit == "month" else d + timedelta(days=1)
    return datetime(nxt.year, nxt.month, nxt.day, tzinfo=ZoneInfo(tz))
src/dataplatform/gold/windows.py:41The window an instant falls in.
def of(t: datetime, unit: str, tz: str) -> str:
    """The window an instant falls in."""
    if unit == "hour":
        return t.astimezone(UTC).replace(minute=0, second=0, microsecond=0).strftime(HOUR_FORMAT)
    local = t.astimezone(ZoneInfo(tz)).date()
    return (local.replace(day=1) if unit == "month" else local).isoformat()

Which windows a run rebuilds

src/dataplatform/gold/windows.py:49The first window of the range: the lookback counted back from the business date.
def range_start(as_of: str, unit: str, lookback: int, tz: str) -> str:
    """The first window the recompute rebuilds on business date `as_of`: `lookback` units back (ADR-006)."""
    d = date.fromisoformat(as_of)
    if unit == "month":
        return _month_add(d.replace(day=1), -lookback).isoformat()
    if unit == "day":
        return (d - timedelta(days=lookback)).isoformat()
    day_end = datetime(d.year, d.month, d.day, tzinfo=ZoneInfo(tz)) + timedelta(days=1)
    return of(day_end - timedelta(hours=lookback), "hour", tz)

For days and months the lookback is counted back from the business date itself. For hours it is counted back from the end of the business date, so a 48-hour lookback covers the business date and the day before it. The table shows the three tyre facts for a run on 22 November 2026, with the business time zone of India. The values are the ones the test table checks.

FactUnit and lookbackFirst window rebuilt
fact_otifday, 352026-10-18
fact_salesmonth, 32026-08-01
fact_scraphour, 482026-11-20T18:00:00Z
src/dataplatform/gold/harness.py:364The full range: from the first window on, plus named windows outside it.
def _range(ctx: Ctx, p: GoldPlan, got: _Inputs, published: Frame | None) -> tuple[str, set[str]]:
    """The windows this run recomputes: the lookback from as_of, plus the windows active plans name and those with
    rows on the unknown member; every window when a withdrawn batch's plan says so (VOID-01)."""
    if p.window is None:
        raise GoldPlanError(f"{p.name}: a fact needs a window")
    first = W.range_start(ctx.as_of, p.window.unit, p.window.lookback, ctx.tz)
    if EVERY_WINDOW in got.forced:  # a withdrawn batch: recompute every window, replace only those that changed
        first = BEFORE_ALL
    return first, (got.forced - {EVERY_WINDOW}) | ctx.engine.windows_with_unknown(p, published)

The range has no upper end. Every window from the first one on is recomputed, including future ones. On top of that, three kinds of window are added even when they are older than the lookback.

Extra windowWhy it is added
A window a correction plan namesA correction to an old order must reach its window however old it is.
A published window with a row on the unknown memberWhen the missing dealer arrives, the row must be given its real key.
Every windowWhen a batch was withdrawn, any window may have held its rows.

Recomputing a window does not mean writing it. The write replaces only the windows whose content changed, as The harness explains.

Completeness: is every input in?

A window is complete when every input of the fact has loaded past the window's end. Until then its rows carry _is_complete = false. The harness does not look at the data to decide this. It compares the window's end with each input's watermark, which is the point in source time up to which silver has taken that input's deliveries.

src/dataplatform/gold/windows.py:65One input has caught up with a window when its watermark is at or after the window's end.
def caught_up(window: str, unit: str, tz: str, watermark: datetime | None) -> bool:
    """One input has loaded past the window's end (ADR-018). No watermark: it has loaded nothing yet."""
    return watermark is not None and end(window, unit, tz) <= watermark
src/dataplatform/gold/harness.py:270A window is complete only if no input is still waited for.
def _complete(ctx: Ctx, p: GoldPlan, window: str, got: _Inputs) -> tuple[bool, list[str]]:
    """ADR-018: every input caught up past the window's end (per-feed inputs: every expected feed delivered it).
    A row the build says still waits for an input (`_input_complete` false) is flagged on its own (finish)."""
    assert p.window is not None
    unit, waits = p.window.unit, []
    for ds, mark in got.marks.items():
        if not W.caught_up(window, unit, ctx.tz, mark):
            waits.append(ds)
    for ds, (feeds, expected) in got.feeds.items():
        owed = {m for m, first in expected.items() if first < W.end(window, unit, ctx.tz)} if expected else set(feeds)
        ok, missing = W.feeds_caught_up(window, unit, ctx.tz, feeds, owed)
        if not ok:
            waits.append(f"{ds} ({len(missing)} feeds, e.g. {', '.join(missing[:3])})")
    return not waits, waits

A worked example

Three orders have promised dates of 9, 10 and 12 November. Silver has taken the order deliveries for 9 and 10 November. The watermark is therefore the end of 10 November. Each row below is one window of the fact.

WindowWindow endsWatermark at or after the end?_is_complete
2026-11-09end of 9 Novyestrue
2026-11-10end of 10 Novyestrue
2026-11-12end of 12 Novnofalse

Inputs that arrive one file per member

The dealer sell-out input is different. It is not one delivery a day. It is one file per dealer per month, and the dealers send them on different days. A single watermark cannot say whether October is in. So ingestion keeps one bookmark per dealer, holding the newest month that dealer has sent, and a month is complete when every dealer who owes a file has sent it.

src/dataplatform/gold/windows.py:70Complete when every expected feed has reached this window. The feeds still missing are returned.
def feeds_caught_up(window: str, unit: str, tz: str, feeds: dict[str, datetime | None],
                    expected: set[str]) -> tuple[bool, list[str]]:
    """A per-feed input (one file per dealer): each expected feed's newest data time falls in this window or a
    later one. Returns (complete, feeds still missing)."""
    missing = sorted(f for f in expected if (t := feeds.get(f)) is None or of(t, unit, tz) < window)
    return not missing, missing
src/dataplatform/gold/harness.py:256Who owes a file: the members of the fact's dimension, each from its first version on.
def _expected_feeds(ctx: Ctx, p: GoldPlan, feeds: dict[str, datetime | None]) -> dict[str, datetime]:
    """The members a per-feed input owes a file for: those of the fact's dimension the feed keys belong to, from
    their first version in the source (a dealer that joined later owes nothing for earlier months). Without such a
    dimension, the feeds seen so far."""
    for d in p.dimensions:
        dim = ctx.tables.gold(ctx.plans[d.name])
        if dim is None or len(d.business_key) != 1:
            continue
        seen = ctx.engine.first_seen(dim, d.business_key[0])
        if set(feeds) <= set(seen):
            return seen
    return {}

The second function answers who owes a file. It looks for a dimension of the fact whose business keys include every feed name. For the sales fact that is the dealer dimension. Each dealer owes files from the start of its first version, so a dealer who joined in November owes nothing for October.

Rows that wait on their own

A model may return a column named _input_complete. A row where it is false is flagged incomplete even when its window is complete. The scrap model uses this for production rows whose inspection result has not been filed.

src/dataplatform/adapters/spark_gold.py:208The flag on each row: the window's flag, and the row's own flag when the model gave one.
    def finish(self, plan: GoldPlan, facts: DataFrame, complete: dict[str, bool], run_id: str) -> DataFrame:
        flags = F.create_map(*[x for w, ok in sorted(complete.items()) for x in (F.lit(w), F.lit(ok))]) \
            if complete else F.create_map().cast("map<string,boolean>")
        done = F.coalesce(flags[self.window_text(plan)], F.lit(False))
        if "_input_complete" in facts.columns:  # a row still waiting for its own input is flagged on its own
            done = done & F.coalesce(F.col("_input_complete"), F.lit(True))
        rows = facts.withColumn("_is_complete", done).withColumn("_run_id", F.lit(run_id))
        return rows.select(*[F.col(c).cast(t).alias(c) for c, t in plan.columns])

Flag or hold

The manifest key completeness.on_incomplete decides what happens to an incomplete window. With flag it is published with the flag false, and a reader or a metric chooses whether to count it. With hold it is not written until it is complete. All three tyre facts use flag.

Late data and restatements

A restatement is a change to a number that readers were entitled to treat as final. Not every change is one. A day that fills up during the day is only filling. The rule is in one function.

src/dataplatform/gold/windows.py:91When a rebuild counts as a restatement.
def is_restatement(before: Numbers | None, after: Numbers | None, window_end: datetime,
                   last_published: datetime | None, waited_for_feed: bool = False) -> bool:
    """A rebuild restates a window when numbers readers were entitled to treat as final change:
        - the window was published complete and its numbers change (a correction, a late change), or
        - the window, already over when it was published, was flagged incomplete because a member's feed had not
          delivered (a dealer's file, LATE-01), and the rebuild that completes it changes its numbers.
    A provisional number still waiting for the ordinary lag of an input (the next hour's lab file), a period still
    open, or a window seen for the first time is not restated. A complete window whose rows all left it is."""
    if before is None:
        return False
    ended = last_published is not None and window_end <= last_published
    seen_final = before.complete or (ended and waited_for_feed)
    if after is None:  # every row left the window (a correction moved them): what readers saw is gone
        return before.complete
    return after.complete and not before.same_numbers(after) and seen_final
Before the runAfter the runRestatement?Why
Not publishedAnythingNoNobody saw an earlier number.
CompleteComplete, different numbersYesA final number changed.
CompleteNo rows leftYesWhat readers saw is gone.
Incomplete, the period was over, a member's file was missingComplete, different numbersYesThe period had ended, so readers took the number as the month's number.
Incomplete, waiting for an input's ordinary delayComplete, different numbersNoThe number was provisional and said so.
IncompleteStill incompleteNoStill provisional.
Incomplete, the period was still openCompleteNoAn open period completing is normal.
IncompleteComplete, same numbersNoOnly the flag moved.

The late dealer file, followed through

In the simulated tyre company, the dealers' October sell-out files arrive over the first five days of the run, and one is late on purpose. In the recorded run of 5 November 2026 at one tenth of full scale, October was flagged incomplete on days 1 to 4. On day 5 the last file arrived, the month became complete, and it was replaced and logged as a restatement: 7,392 rows became 9,404 and 602,101 units sold out became 765,689. On day 6 a dealer resent a file with the same content. Nothing was replaced and nothing was logged. The table is in docs/W1D-GOLD.md.

src/dataplatform/gold/harness.py:453What is written to the run log about corrections, restatements and completeness.
def _record(ctx: Ctx, p: GoldPlan, got: _Inputs, restated: list[Restatement], incomplete: list[str],
            why: dict[str, list[str]], counts: dict[str, int]) -> None:
    for b in got.applied:
        _step(ctx, p, "detect.HIST-05", "warned", {"windows": len(b.windows), "corrections": got.corrections},
              f"correction plan {b.plan_id} applied from {b.activate_on}: windows {', '.join(b.windows)} rebuilt "
              f"({b.reason[:200]}; recorded by {b.recorded_by})")
    plans = ", ".join(b.plan_id for b in got.applied)
    for r in restated:  # RSTM-01: one structured record per changed window, read by ops_restatements
        reason = f"correction plan {plans}" if r.detector == "HIST-05" else "late data"
        nums = {f"{k}.before": v for k, v in r.before.items()} | {f"{k}.after": v for k, v in r.after.items()}
        _step(ctx, p, RESTATEMENT_STEP, "warned", nums, f"window={r.window}; reason={reason}; detector={r.detector}")
    for det in ("LATE-01", "HIST-05"):
        mine = [r for r in restated if r.detector == det]
        if mine:
            _step(ctx, p, f"detect.{det}", "warned", {"windows_restated": len(mine)},
                  "restated: " + "; ".join(f"{r.window} {r.before} -> {r.after}" for r in mine)[:3900])
    waits = sorted({w for ws in why.values() for w in ws})
    _step(ctx, p, "gold.completeness", "succeeded", {"windows_incomplete": len(incomplete)},
          (f"flagged incomplete: {', '.join(incomplete[:12])}{' ...' if len(incomplete) > 12 else ''}; waiting for "
           f"{'; '.join(waits)[:1500]}") if incomplete else "every window complete")
    _step(ctx, p, "gold.write", "succeeded", counts)

Each restated window gets one gold.restatement row with every number before and after. The operations layer reads those rows into a table of restatements with one row per window and measure. Operations covers that table.

Corrections

Sometimes a value in the source was simply wrong and someone corrects it after the fact, for example a promised date that was keyed in a day early. The platform does not edit silver. The correction is recorded in the metadata store, and the gold harness lays it over the silver rows in memory before the model runs.

Corrections come in a plan. A plan names the dataset, the date from which its corrections are active, the windows it expects to change, a reason, and who recorded it.

src/dataplatform/meta/port.py:120A plan and one correction, as the metadata store hands them to the harness.
@dataclass(frozen=True)
class BackfillPlan:
    """A plan that activates corrections on a date and names the gold windows to rebuild (ADR-017, DPC-0082)."""

    plan_id: str
    source: str
    dataset: str
    windows: tuple[str, ...]  # the windows the recompute rebuilds when the plan activates
    activate_on: str  # business date (ISO) from which its corrections are active
    reason: str
    recorded_by: str


@dataclass(frozen=True)
class Correction:
    """One corrected value (DPC-0082: entity, field, value, applies from and to, who, why, status). The entity is the
    plan's dataset; who and why are the plan's. `key` holds the business key columns and their values."""

    key: dict[str, Any]
    field: str
    value: str | None
    was: str | None = None
    applies_from: str | None = None  # business dates; None = always
    applies_to: str | None = None
    plan_id: str = ""  # filled by the store
    status: str = "recorded"  # recorded until the plan's date, then active (filled by the store)
src/dataplatform/gold/harness.py:211Which corrections are applied to one input in this run.
def _corrections(ctx: Ctx, p: GoldPlan, dataset: str, active: list[BackfillPlan]) -> tuple[list[Correction], int]:
    """Active corrections for one input. When a correction moves a row between windows (it corrects the fact's own
    window column), it is applied only if its plan names both windows: a correction never restates a window by
    surprise (ADR-017). Those it cannot apply are counted."""
    source, name = dataset.split(".", 1)
    named = {b.plan_id: set(b.windows) for b in active}
    out, skipped = [], 0
    for c in ctx.store.corrections(source, name, ctx.as_of):
        if c.status != "active" or c.plan_id not in named:
            continue
        moves = p.window is not None and c.field == p.window.column
        if moves and not {str(c.was), str(c.value)} <= named[c.plan_id]:
            skipped += 1
            continue
        out.append(c)
    return out, skipped

The rule in the middle is the important one. If a correction changes the fact's own window column, it moves a row from one window to another. It is applied only if the plan names both the old window and the new one. Otherwise it is skipped and counted in corrections_not_applied. A correction can therefore never change a window that the person who approved the plan did not expect to change.

src/dataplatform/adapters/spark_gold.py:145Laying corrections over silver rows: a small table of fixes joined on the business key.
    def overlay(self, frame: DataFrame, corrections: list[Correction], keys: tuple[str, ...]) -> DataFrame:
        types = {f.name: f.dataType.simpleString() for f in frame.schema.fields}  # not .dtypes: not on every engine
        for fld in sorted({c.field for c in corrections}):
            mine = [c for c in corrections if c.field == fld]
            fields = {f.name: f.dataType for f in frame.schema.fields}  # StructType[name] is not on every engine
            schema = T.StructType([*[T.StructField(k, fields[k]) for k in keys],
                                   T.StructField("_fix", T.StringType())])
            fixes = self.spark.createDataFrame([(*[c.key[k] for k in keys], c.value) for c in mine], schema)
            # shiftleft: allow SL-COST-CODE-14 -- joined on the business key columns `keys` (a USING join)
            frame = (frame.join(fixes.hint("broadcast"), list(keys), "left")
                     .withColumn(fld, F.coalesce(F.col("_fix").cast(types[fld]), F.col(fld))).drop("_fix"))
        return frame

In the recorded run, a planted correction to promised dates became active on day 11. The plan was applied on that day, the windows from 13 to 18 November were restated, and no window outside the plan changed.

A withdrawn batch

A batch that turns out to be bad after it was published can be withdrawn. Withdrawing is also called voiding. It marks the batch void, so no reader takes it again, and records a plan whose window list is a star, meaning every window.

src/dataplatform/recovery.py:18Withdrawing a batch: void it, and record a plan that names every window.
def withdraw(store: MetadataStore, source: str, dataset: str, batch_key: str, *, reason: str, approved_by: str,
             today: date) -> BackfillPlan:
    store.void_batch(source, dataset, batch_key, reason, approved_by)
    plan = BackfillPlan(f"void-{batch_key[:40]}", source, dataset, (EVERY_WINDOW,), today.isoformat(),
                        f"VOID-01: batch {batch_key} withdrawn: {reason}", approved_by)
    store.record_corrections(plan, [])
    return plan
  1. Withdraw. The batch is voided with a reason and the name of who approved it.
  2. Rebuild silver. Silver for that dataset is rebuilt from bronze. The reader skips the void batch, so its rows are gone.
  3. Run gold as usual. Every fact that reads the dataset sees the plan, sets its range to every window, recomputes all of them, and replaces only the ones whose content changed.
  4. Read the log. Each changed window that had been complete is logged as a restatement whose reason names the plan.

No row is deleted by hand at any point. The withdrawn rows leave gold because the rebuild no longer produces them.

The settings that drive it

WhereKeyWhat it meansValues in the tyre example
Fact manifestwindow.columnThe column a fact is sliced by.promised_date, sale_month, cure_hour
Fact manifestwindow.unitThe size of one window.day, month, hour
Fact manifestwindow.lookbackHow many windows back each run recomputes. Default 0.35, 3, 48
Fact manifestcompleteness.on_incompletePublish an incomplete window flagged, or hold it back. Default flag.flag
Fact manifestcorrectionsWhether corrections are laid over the inputs. Default false.true, true, false
Domainbusiness_timezoneWhere a day and a month start.Asia/Kolkata
Runthe business dateThe date the run is for. The range is counted back from it.one per day
Metadata storesilver's bookmark and the file logThe watermark of each input.
Metadata storeper-feed bookmarksThe newest month each dealer has sent.

The date dimension

Questions about windows usually need a calendar: which fiscal quarter a day is in, whether it was a working day. No source sends one, so the platform generates it. It is one row per day, keyed by the date and by a number such as 20261113. A fact can join it on any of its date columns without any change to the fact.

src/dataplatform/gold/calendar.py:26The fiscal year, quarter and month of a day, for a fiscal year that starts in a given month.
def fiscal(d: date, start_month: int) -> tuple[int, int, int]:
    """(fiscal year, fiscal quarter, fiscal month) for a fiscal year starting on the 1st of start_month; the fiscal
    year is named by the calendar year it ends in (India: April 2026 to March 2027 is fiscal year 2027)."""
    if not 1 <= start_month <= 12:  # noqa: PLR2004 -- months
        raise ValueError(f"fiscal start month {start_month} is not 1 to 12")
    shifted = (d.month - start_month) % 12
    year = d.year + (1 if start_month > 1 and d.month >= start_month else 0)
    return year, shifted // 3 + 1, shifted + 1

The fiscal year is named by the calendar year it ends in. With a start in April, 1 April 2026 is the first day of fiscal year 2027, and 14 November 2026 is in fiscal quarter 3, fiscal month 8.

src/dataplatform/gold/calendar.py:36One row per day. Holidays are a list the caller passes in. Nothing is guessed.
def calendar_rows(start: date, end: date, fiscal_start_month: int = 4,
                  holidays: Iterable[date] = (), weekend: tuple[int, ...] = (5, 6)) -> list[dict[str, Any]]:
    """One row per day from start to end inclusive. `weekend` holds weekday numbers (Monday = 0)."""
    if end < start:
        raise ValueError(f"calendar end {end} is before start {start}")
    off = set(holidays)
    rows = []
    d = start
    while d <= end:
        iso_year, iso_week, _ = d.isocalendar()
        fy, fq, fm = fiscal(d, fiscal_start_month)
        is_weekend, is_holiday = d.weekday() in weekend, d in off
        rows.append({
            "date_key": d.year * 10000 + d.month * 100 + d.day, "date": d, "year": d.year,
            "quarter": (d.month - 1) // 3 + 1, "month": d.month, "month_name": MONTHS[d.month - 1],
            "year_month": f"{d.year}-{d.month:02d}", "iso_week": iso_week, "iso_year": iso_year,
            "day_of_month": d.day, "day_of_week": d.weekday() + 1, "day_name": DAYS[d.weekday()],
            "is_weekend": is_weekend, "is_holiday": is_holiday, "is_working_day": not (is_weekend or is_holiday),
            "fiscal_year": fy, "fiscal_quarter": fq, "fiscal_month": fm,
        })
        d += timedelta(days=1)
    return rows

The run script writes the calendar whole on every run, for five years either side of the simulated company's start date, with the fiscal year starting in April.

How it is tested

The window arithmetic, completeness and the restatement rule are pure functions, so tests/dataplatform/test_gold_plan.py tests them as tables with no engine. tests/dataplatform/test_gold_harness.py then runs the late file, the correction and the withdrawn batch through the real harness. tests/dataplatform/test_gold_calendar.py covers the calendar, and tests/dataplatform/test_ops_restatements.py covers the table of restatements.

tests/dataplatform/test_gold_plan.py:203The restatement rule as a table: numbers before, numbers after, when it was last published, whether a feed was waited for, and the verdict.
        ("first publication", None, N(5, (("q", 1.0),), True), AFTER, False, False),
        ("same numbers", N(5, (("q", 1.0),), True), N(5, (("q", 1.0),), True), AFTER, False, False),
        ("a complete window changes (a correction)", N(5, (("q", 1.0),), True), N(5, (("q", 2.0),), True), AFTER,
         False, True),
        ("a late member's file completes an ended month (LATE-01)", N(5, (("q", 1.0),), False),
         N(6, (("q", 2.0),), True), AFTER, True, True),
        ("an input's ordinary lag completes an ended hour", N(5, (("q", 1.0),), False), N(5, (("q", 2.0),), True),
         AFTER, False, False),
        ("a provisional number of an ended month changes", N(5, (("q", 1.0),), False), N(6, (("q", 2.0),), False),
         AFTER, True, False),
        ("an open period completes", N(5, (("q", 1.0),), False), N(6, (("q", 2.0),), True), BEFORE, True, False),
        ("only the flag moves", N(5, (("q", 1.0),), False), N(5, (("q", 1.0),), True), AFTER, True, False),
        ("every row moved out of a complete window", N(5, (("q", 1.0),), True), None, AFTER, False, True),
        ("every row moved out of a provisional window", N(5, (("q", 1.0),), False), None, AFTER, False, False),
    ],
)
def test_is_restatement(case, before, after, published, feed, restated):
    assert W.is_restatement(before, after, END, published, feed) is restated
tests/dataplatform/test_gold_harness.py:274Dealer D-1 sends October on day 1 and D-2 on day 5. The month is incomplete, then complete and restated from 5 units to 12. A resend changes nothing.
def test_a_late_member_file_completes_and_restates_the_month(monthly):
    monthly.silver("feed", FEED, [{"dealer_code": "D-1", "month": OCT, "units": 5}])
    feed(monthly.store, DAYS[0], ["D-1"])
    day1 = monthly.run(DAYS[0], only={"dim_dealer", "fact_monthly"}).by_name()["fact_monthly"]
    assert day1.incomplete == ["2026-10-01"] and not day1.restated
    assert "src.feed (1 feeds, e.g. D-2)" in next(s["message"] for s in monthly.steps("gold-1")
                                                  if s["step"] == "gold.completeness")
    monthly.silver("feed", FEED, [{"dealer_code": "D-1", "month": OCT, "units": 5},
                                  {"dealer_code": "D-2", "month": OCT, "units": 7}])
    feed(monthly.store, DAYS[4], ["D-2"])
    day5 = monthly.run(DAYS[4], only={"dim_dealer", "fact_monthly"}).by_name()["fact_monthly"]
    assert day5.incomplete == []
    assert [(r.window, r.detector) for r in day5.restated] == [("2026-10-01", "LATE-01")]
    assert any(s["step"] == "detect.LATE-01" for s in monthly.steps("gold-2"))
    [rst] = [s for s in monthly.steps("gold-2") if s["step"] == "gold.restatement"]  # RSTM-01
    assert "window=2026-10-01; reason=late data" in rst["message"]
    assert (rst["counts"]["units.before"], rst["counts"]["units.after"]) == (5, 12)
    resend = monthly.run(DAYS[5], only={"dim_dealer", "fact_monthly"}).by_name()["fact_monthly"]
    assert (resend.replaced, resend.restated) == ([], [])
tests/dataplatform/test_ops_restatements.py:18From run log rows to the table of restatements: one row per window and measure, with the change.
@pytest.mark.parametrize(("case", "steps", "want"), [
    ("late data moved a published day",
     [step("gold.restatement", {"rows.before": 10, "rows.after": 12, "qty.before": 100.0, "qty.after": 130.5},
           "window=2026-11-10; reason=late data; detector=LATE-01")],
     [("2026-11-10", "late data", "qty", 100.0, 130.5, 30.5), ("2026-11-10", "late data", "rows", 10.0, 12.0, 2.0)]),
    ("a correction plan", [step("gold.restatement", {"rows.before": 5, "rows.after": 5},
                                "window=2026-11-02; reason=correction plan P-7; detector=HIST-05")],
     [("2026-11-02", "correction plan P-7", "rows", 5.0, 5.0, 0.0)]),
    ("other steps are not restatements", [step("gold.write", {"rows": 3}, "")], []),
])
Not built yet, or known gaps
  • Completeness only knows deliveries by business day. An input that silver loads several times a day still counts as loaded only through the end of the newest whole day it has taken.
  • Whether an input arrives one file per member is inferred from ingestion's bookmarks. No manifest key declares it.
  • A day on which each dealer's file is due is not known to the platform. October is flagged until the last dealer has sent, with no notion of a file being overdue.
  • The specification says restatements are logged but not when a rebuild is one. The rule on this page is a decision taken in the code.
  • A restatement caused by a withdrawn batch is recorded under the same detector name as a correction plan. Only the reason text tells them apart.
  • Restatements live in the run log. There is no separate restatement table in the metadata store.
  • The date dimension has no holidays for the tyre company. The run script passes none.
  • Plant time zones are not declared anywhere the platform reads, so an hour window is always a UTC hour.